Lessons in Failure: Removing the Old Debugger from Firefox

Joining the Firefox DevTools team was a dream come true for me.  I shared why in the first episode of the Script & Style show but the quick summary is that I grew up loving Firefox (even before it was called "Firefox") because Mozilla cared about developers via their early developer tools.  If you told 18 year old David that he'd one day work at Mozilla and would eventually find his way to the Firefox DevTools team he wouldn't have believed you.  I've learned so much over the past eight months and I've accepted every challenge to continue learning more about Firefox, DevTools, and web technologies.

Two months ago I was approached about taking on the task of removing the old debugger from Firefox.  The "new" debugger (which you should totally contribute to!) has been Firefox's default debugger for roughly one year now and continues to draw accolades from the developer community, so it was the right time to remove the older, stale debugger which was available behind a preference in about:config.  I gladly accepted the task because it would help me learn more about mozilla-central (i.e. the Firefox source code) and was a testament to the quality of work that Mozilla employees and the new debugger's contributors that there was so much confidence in the new debugger.  It was also a chance to show my quality and earn some credibility amongst the larger DevTools team.

While I've completed the task, I look back on my performance and methodology as an embarrassing, overly complex failure in so many ways.  While I can't go back and start the process over, I am big enough to look back on the process and realize what I did wrong and what I will do better in the future.

The Setup

• The old debugger was written in XUL and was only available via a Firefox preference in about:config
• The old debugger's usage numbers were very tiny
• The old debugger was not under active development
• The old debugger lived in Firefox's mercurial repository called "mozilla-central"
• The new debugger lived within the old debugger's directory, in a subdirectory called new
• The old debugger included dozens of unit tests, with roughly 20 tests that tested the debugger's server, and so those tests couldn't simply be deleted away

Removing the Debugger: Attempt 1

My first attempt at removing the old debugger happened on a Friday and took only three hours.  I deleted every old debugger file, removed the preference, and elevated the new debugger to the old debugger's spot.  Hell, this task is way less difficult than it seemed!  Everything worked great -- job done, right?!

Wrong...soooooo wrong.  I deleted every old test (I wasn't aware we needed to keep some), I didn't remove all unused image assets that lived in another directory (Mozilla has a tool to find these unused assets), and it was simply the most naive approach I could have taken.

This patch was obviously and rightfully rejected but I wasn't too upset about it: I hadn't spent much time on it and it gave me a quick introduction to parts of a devtools' internals that I hadn't known about before.

Removing the Debugger: Attempt 2

My second attempt to remove the old debugger saw me start out by shifting some of the old debugger's tests (tests still relevant because they tested the debugger server) to another location.  There was an open bug that listed those tests and so I spent a week moving those tests over.  The act of updating and moving those tests wasn't difficult but it was time-consuming, as I was rediscovering my mercurial skills and trying to not to mess it all up -- I didn't want to start over again.

I submitted a series of patches (nearly 20!) which methodically moved old tests to their new place while deleting those that weren't needed.  I felt OK about how things were going but I was starting to feel the heat (from myself) about how long the task was taking.  I also felt like I was letting the debugger team and community down because I wasn't as available to review others' code, fix bugs, and mentor as much as I normally had.

Once I'd moved all the cited tests, I submitted the last patch and was done, right?  Nope.  More old-but-relevant tests were discovered that we needed to keep, and a senior person on the team mentioned that removing the old debugger and elevating the new in a single commit would creating history problems when there were duplicated file names between the old directory and the new.  For example, if the old directory had a head.js file and the new directory also had a head.js file, it may look like the old directory's head.js file simply changed, and the history of the new head.js would be gone.  It was a very valid point that I simply hadn't thought about.

F*ck.

Removing the Debugger: Attempt 3

In speaking with the senior DevTools developer that reviewed the patch, he casually said something that had a massive impact on me:  I could remove the old debugger without removing/moving the old tests (very temporarily skip them) while still leaving the new debugger in its current directory.  This makes sense because the larger utility for Firefox users was a smaller Firefox.  Skipping tests is never ideal but the majority of them were essentially useless since the old debugger wasn't used anymore.

With that in mind, I removed all of the old debugger assets in one patch, then quickly thereafter went about moving old debugger tests and re-enabling them.  It was a strategy that relieved massive pressure and finally I could say "the old debugger is gone".

There were a few more "sub-attempts" that led me astray along the way, but these were the most relevant and concise.

Feeling the Failure

I've never felt the feeling of failure so hard as this experience; not even close.  The first failure didn't hurt much because I didn't spend much time on it and I knew, even while doing it, it was probably overly aggressive and it just felt too easy.  The second failure hit me like a Mike Tyson punch.  Migrating dozens of test to the new API Was incredibly time-consuming and feeling like I had completed them all felt like I'd finally climbed the mountain.  Being told there were further tests to migrate was the straw that broken the camel's back.

The task got to the point of consuming me. I started bringing my laptop to family dinner every night.  I had a hard time sleeping at night.  I tried getting away from it for a few hours but my mind wouldn't let it go.  I COULDN'T LET IT GO.  The first tab in browser every fucking morning was Bugzilla bug 1314057 -- I hated that fucking tab.  I'll never forget that bug number for the rest of my night.  Every weekly DevTools meeting I teased that removing the old debugger was almost done, only to feel like a clown the next week when a patch was rejected or there were more tests to move.  I tried to show a brave, professional face whenever asked about the task's status but privately I felt humiliated, stupid, any but a "senior developer."

I continued digging myself into a dark, depressing foxhole, going to daily war against myself.

To the credit of my manager and the rest of the DevTools team, they were all incredibly supportive and repeatedly told me they understood the complexity of the task and that I should be proud of what I've learned, the work I had completed, and the impact it would have.  Convincing a broken, self-loathing man of that was an impossible task though.

Learning from Failure

Having the luxury of looking backward, I can honestly say I did everything wrong.  Everything.  It's easy for me to sum it up in one word but let's analyze what led me off of the cliff.

Insufficient Communication and Planning

One of my favorite qualities about myself is that, even at an ancient 35 years old, I still have the childlike enthusiasm to attack new tasks without fear.  My father once told me he tried teaching me how to dive off the diving board at age 3 and had to frantically dive in to save me because I dove in before he was ready.  I'm still excited to jump on something...but it leads me to making mistakes because I don't have a solid enough plan.

Before I did anything actionable, I spoke to one colleague about how to go about removing the old debugger.  The colleague I spoke to is incredibly smart but a task this big really called for speaking to three or four different people.  Had I spoken to the senior DevTools developer at the beginning, and worked on removing assets first, I would have saved weeks of time.

Why didn't I speak to more people?  The truth is that through eight months of being on the DevTools team I had been isolated to only working on the debugger, so I was timid when it came to interacting outside my team...and it cost me dearly.  I'll never attack a problem like this again without talking to more people in the future.  I had a wealth of people around me but being timid cost me so much mental anguish.

"Get Your Mind Right, Get Your Grind Right"

Ice Cube's wisdom rings in my head when I think about how I mentally lost the plot with this task.  This task took over my life for a month and the seething and self-loathing made the task bleed on.  I desperately need to learn to compartmentalize my work; I need to be able to shut my mind off to one task so that I can focus on others, if only for a few hours.  At 35 years old I'm feeling hopeless to a mental change but it's something I need if I want to earn promotion at Mozilla or another employer.  Oh, and for my own sanity too.

Advocate For Myself

Instead of trying to be an iron-faced superhero, my idea of being professional, apparently, I should have spoken up sooner to my manager or the larger team.  The more desperate and negative I felt, the further I sunk into my dark hole instead of reaching for help or more ideas.  I didn't consciously avoid asking for help though -- I always felt like I was on the cusp of breakthrough, despite it never happening.  I also didn't purposely avoid asking for advice at the beginning -- I just ... I didn't see the need for additional help.

Battered and Bruised, I Lick My Wounds

I'd call this experience "humbling" but that word doesn't adequately express the self-hatred and darkness I found myself in.  It's one thing to have impostor syndrome and it's another to question how you were ever considered a senior developer fifteen years into your career.  It's going to take a long time to build myself back up from this experience.  This wont be the last time I doubt myself but it's certainly going to be the last time I make the bush league mistakes I did with removing the old debugger.  Mentally, emotionally, and physically...I can't put myself through this again.

The show goes on though.  I'm off to races on my next task, striving to do better!