Last week, I wrote in some detail about some of the specifics of how the Federal healthcare portal seems to have violated basic principles of good software delivery. Now I want to talk a bit about the more general factor that I think led to all those cut corners and shoddy deliverables.
One of the oldest and shortest jokes in the computer industry is “Time. Money. Features. Pick any two.” To some extent, you can swap quality out for features, because poorly delivered functionality is essentially non-existent functionality. In almost all commercial software projects, time and money both end up being the parts that give in the end. In other words, the original feature set almost never gets cut, but the project goes over budget and delivers late.
The larger the project, the more true the law tends to be. In spite of all the best software practices, including Agile (which, allegedly, healthcare.gov was developed under), software development of large and complex systems tends to be an imprecise art, more research than manufacturing. I’ve worked on a $50 million, 3-year project at a major financial house that eventually turned into a $70 million, 5-year project. Add in the overhead and red tape of government project management, and it can only get worse.
So now let’s look at the healthcare.gov situation. It almost certainly had a fixed budget, a set of features that it absolutely had to deliver, and to complete the perfect storm, a drop-date delivery date that could not be changed in the slightest. All three variables, locked in stone. Something had to give, and that something was quality.
When a project is running late, code reviews become rushed, QA gets deferred, P1 bugs get reprioritized as P2s. And while most of the back end integration work was probably tested well and early (since you can test system-to-system integration with unit-level or shimmed code), most of the UI and end-to-end testing probably had to wait until the very end, when the whole system was up and running. And apart from the overload failures we saw (and continue to occasionally see), most of the errors appear to be occurring on the client side, specifically in the JavaScript, which was likely the last thing to get any love from QA.
Perhaps more resources might have helped, but another koan of the trade states that “adding resources to a late project only makes it later.” Past a certain point, all that new bodies will do is create overhead for the already stressed-out staff, getting the new folks up to speed.
There’s a phrase you hear in aviation, “the coffin corner.” It’s a part of the performance chart of a high-speed aircraft where the combination of speed and altitude can make any maneuver stall the aircraft, perhaps irrecoverably. Software projects can find themselves in the same kind of dilemma: too close to deadline to add more people, too committed to a feature set to drop anything, and unable to slip the date. When that happens, buckle your seat belts, put up your tray table, and prepare for a nose-dive.
Editor’s Note: If you’re interested in doing your part to improve the U.S. health care system and want to know more, get two free ebooks to spark your thinking: “Hacking Healthcare” and “Open Government.”
[adrotate banner=”4″]