Dither and Bicker

It seems most of my development career has been spent on either propping up a legacy project, or writing a green fields project to dethrone a legacy project. In nearly every case the green fields project is lauded as necessary. Customers are said to be longing for a completely new version. The legacy product is slow, and ugly. Competitors mock it.

Then reality intrudes.

Every green fields project, should it even get released at all, will be a step down in terms of functionality from its legacy forebear. At best it will have a new feature or two that the previous software couldn’t achieve gracefully. Whatever new is offered, customers will ultimately be asked to sacrifice some of the previous feature-set and will need to learn a new workflow.

Customers want to focus on their primary business. If they’re paying your company for a product its typically because its not in their interests to develop that functionality themselves. Once they’ve admitted that the need to pay for 3rd party software and pay the costs of training their employees to manage it, they often have precious little interest in ramping up on new version of the software. In practice a more-modern interface is rarely worth shelling out thousands of dollars while disrupting workflows. Acme Anvil Co wants to churn out anvils, not experiment with new payroll systems.

I’ve seen my share of green fields projects die completely. Some never make it to a public release, some get pulled as soon as the market passes its judgment. Having watched all this unfold at my current employer again, I want to focus on a green fields project that did make it and why it was successful.

I was an early employee at a mobile advertiser. The initial adserver was written fast, dirty, and in PHP. It had to function well enough to prove to potential funders and clients that we were a real operation that could provide value. By these standards it succeeded.

A few years into the company’s lifespan management decided for a full rewrite, changing nearly everything. The green fields ad server would be in Java and would be designed to scale.

The best two developers were tasked with this rewrite. The company certainly felt their absence. New features languished and the legacy ad server accumulated bugs.

When the Java ad server was finally ready, the CTO made a show of the rollout. One employee, Katie, had been filling in for the legacy ad server’s gaps by manually scheduling campaigns. So a head-to-head competition was scheduled: Katie vs the new Ad Server, who could schedule the campaigns more efficiently?

We all decamped to a nearby bar where the CTO revealed that Katie had won. Everyone congratulated her while the CTO paid for all our drinks. Within a few months the Java ad server had closed the gap. The PHP ad server was decommissioned without incident. Katie was free to focus her energies on parsing through ad click rates and eCPM rather than manually scheduling thousands of campaigns. The green fields ad server wouldn’t need a significant rewrite again, surviving and handling increasing scale beyond the company’s eventual IPO.

So what’s the moral here? Why did this green fields project go right?

Most importantly, the CTO made the investment at the right time. Every day legacy software persists, it gets harder to replace. Next the architects made the right choices in terms of languages. PHP is perfectly acceptable, ideal even, for many types of applications. The PHP ad server did its job, after all. It allowed the company to ramp up quickly and function while a longterm option was getting phased in. A persistent, long-running process though is a stretch for PHP. Java is better suited for that type of software. A decade later, Java still would be a good choice for this type of application. The developers resisted the urge to use a boutique or up-and-coming tech solution and instead went down the boring road of using what they knew would work.

I often think about the “Katie vs Ad Server” competition. This was a great bit of leadership. It created a safe space where the new application could breathe and softened expectations. I’ve seen too many leaders overreact when a green fields project doesn’t immediately deliver the promised results. They’ll change corporate hierarchies, move to a different interpretation of “agile”, or bring in experienced industry experts to reorientate development. In this case however the early stumbles were understood to be a healthy and natural part of the maturation process. A great bonus was that the whole company got to celebrate one of their coworkers at the same time.

In summation leadership:

• Identified the need to replace legacy software early, really before the feature set had a chance to balloon.
• Still left the business with a product they could sell
• Slowed feature-development to deliver the product
• Maintained a realistic view of the initial release
• Didn’t over-promise the initial release, and didn’t overreact to early problems

And the developers:

• Made a wise decision in technology and resisted the urge to incorporate unproven new technologies
• Understood the problem they were being asked to solve
• Focused on writing a solid foundation that could be built out in future releases.

Given what I’ve seen elsewhere in my career, this is an absolute best case scenario. There was good leadership, talented developers, and a good company culture to support the rewrite. Don’t mistake me either, there was nothing inherently special about the company that allowed the green fields project to succeed. Things changed there too when they took the wrong lessons from their success. This opened up a period where “Java was the answer” and all of our applications were slated to be rewritten in Java. If the old LAMP (Linux, Apache, MySQL, PHP) stack was ill-suited for a heavy weight ad server, it was still acceptable if not ideal for web sites. The rewrites for these components were no better considered than “Java is inherently better than PHP and so whatever we write in Java will be an improvement.” I didn’t stick around to see that one through but I know it didn’t end well and after floundering for a couple years the company was back to using the LAMP stack.