What Developers Can Learn from Healthcare.gov

Remember, even a failure can serve as an example of what not to do

The first highly visible component of the Affordable Health Care Act launched this week, in the form of the healthcare.gov site. Theoretically, it allows citizens, who live in any of the states that have chosen not to implement their own portal, to get quotes and sign up for coverage.

I say theoretically because I’ve been trying to get a quote out of it since it launched on Tuesday, and I’m still trying. Every time I think I’ve gotten past the last glitch, a new one shows up further down the line. While it’s easy to write it off as yet another example of how the government (under any administration) seems to be incapable of delivering large software projects, there are some specific lessons that developers can take away.

1) Load testing is your friend. If there’s a positive message that we can glean from the collapse of the portal, it is that there are a LOT of people interested in getting healthcare via the government. Unfortunately, that has led to what is effectively a DDoS attack. It has become abundantly clear that the site was never stress-tested under anything like the type of load it is encountering. The solution so far has been to put people into a queue, something that would get a site like Amazon laughed out of the marketplace. “I’m sorry, we’re a little busy right now, try shopping later?!!”

Creating realistic load testing of a site as complicated as healthcare.gov isn’t easy, but just having a thousand bots load the home page isn’t going to give you a realistic load test, especially of database transactions. You need to really bang on the core functionality of the site, and tune the heck out of it.

2) Pretty doesn’t trump functional. The site is very well designed from a graphical perspective, and is clearly using lots of Javascript and AJAX to do snazzy transitions and requests in the background. Unfortunately, it doesn’t seem to be interacting with the intermittent failures on the backend very well. If you’re going to make requests behind the scenes, you need to be very tolerant of failures. The healthcare.gov site seems to fail silently and leave a broken user experience in its wake, with no way to continue. Nothing drives a user crazy more than having to go through the same form over and over because of failures that leave them high and dry.

3) Validation logic needs to actually work. One of the more interesting early failures of the site was that the username required numbers, but the instructions made no mention of it (and neither did the failure message). It’s especially easy to get an impedance mismatch when you have both client Javascript and server code trying to do the same validations. Keeping the client code, server code, error messages and instructions in sync can be very difficult, but getting it wrong will lead to enraged users.

4) UX isn’t a luxury. User Experience is a very precise art, and when you have a site with as many moving parts and multiple paths as healthcare, making sure that the user always knows where they are, and what the next step is, is key. There have been several times during the sign-up process where I was left in a deathtrap of UI I couldn’t escape from, and it was unclear what the next step was.

The biggest takeaway though, is that the way that the federal government bids out software is fundamentally broken. There are clearly companies in the industry who understand exactly the kind of problems that healthcare.gov needed to address. Intuit’s online TurboTax is much more complicated than the sign-up process for healthcare, and it works under heavy load. Amazon and Google both handle crushing loads gracefully as well. Why can’t the government draw on this kind of expertise when designing a site as critical to the public as healthcare.gov, rather than farming it out to the lowest bidder?

Editor’s Note: If you’re interested in doing your part to improve the U.S. health care system and want to know more, get two free ebooks to spark your thinking: “Hacking Healthcare” and “Open Government.” You also may be interested in James’ follow-up piece, “You Can’t Legislate Away the Time, Money and Features Law.”


Sign up for the O'Reilly Programming Newsletter to get weekly insight from industry insiders.
topic: UI/UX
  • kgelner

    ‘If there’s a positive message that we can glean from the collapse of the portal, it is that there are a LOT of people interested in getting healthcare via the government.”

    How can you say that without knowing the load that caused it to collapse? It could have been ten people.

    Near as I can tell the only people signing up so far are journalists, bloggers and OCA volunteers (although the last ones are lying when they say they actually signed up).

    • ToBeFair

      To be fair, the white house did release the following numbers:

      • glitch31

        The same WH that said it would be transparent, that ACA would make premiums cheaper, supports Muslims, hates Christians, won’t negotiate, spends money to get out of debt, and won’t tell us what happened concerning bin Laden; Benghazi, Fast and Furious; IRS targeting of those who disagree with the administration; and keeps telling us the economy is great based on the number of new jobs being created but neglects to state most of them are only part-time positions?

        Yeah, sure we can trust what they tell us.

      • bee-rye

        Notice the number that’s missing — the number of people who actually enrolled in a plan. Leaving that out suggests that an embarrassingly small number of people were actually able to enroll.

        • MJ12

          Or even *wanted* to enroll after seeing the increase in premiums and what the outrageous annual deductibles would be.

          • JenniferPaige

            Actually, there’s a huge variety of plans to chose from and the premiums are really low on a lot of them. As well as the deductibles. Our premium (my husband and I) is 1/4 of what it was when we looked into getting insurance last year. And our deductible is ZERO. We actually enrolled – we got the lowest cost Silver Plan which was $121 a month with a $0 deductible and and a 30% copay. All basic services such as colon screenings, mammograms, etc etc are completely free.

          • scarymatt

            You are clearly a magical being. I am apparently one of the unlucky people who have an orphaned account from the great password reset, so I created a new one. Now when I log in, I don’t even go to the waiting room. I just get a blank page. That’s probably more honest than promising me something they can’t deliver.

            It’s amusing to see the WTFs evolve.

          • JenniferPaige

            Well, it WAS a pain in the ass to get to it lol. We started at 6am the day it was first available and were finally able to get a plan 3 days later at midnight. I still get a blank page when I log in sometimes but we already have our plan set. It seems like the log in is faster today though. I just don’t think anyone should be yelling about high deductibles and premiums when they haven’t even seen what they are yet.

          • Garys

            Jennifer: I’m sorry, but I highly doubt you. No one seems to have had the experience you describe.

          • JenniferPaige

            I’m sorry to hear you don’t believe me Garys. I’m looking for a way to upload a photo to my Disqus comment because I made a screenshot when I was finished with the enrollment process that shows it was completed. I’m not seeing a way to upload a photo in a reply however. The only problem my husband and I are having right now is that we declined dental care but the website keeps telling us we need to pick a dental plan. It doesn’t seem to be saving our information on that. But we DID finish the entire enrollment – at the end of it, it told us what the health care plan was we had chosen and asked for electronic signatures to verify it. DONE. My theory is this – the people who have enrolled aren’t on the websites complaining. The people who are having issues are all on here venting. It presents a skewed picture as far as how many have been able to enroll.

  • Alexander Mohr

    Why isn’t it better? I’d bet its because its having a hard enough time just staying open, just look at the shutdown. So many cutbacks from the original program, I’m amazed it even made it this far.

    • Joe Eckstein

      While the deteriorating income thesis is valid… that’s how government ALWAYS works, sadly. All they are doing is shuffling OPM around – Other Peoples Money.

  • Joe Eckstein

    Even TaxSlayer which probably does quite a large volume by now is smooth. I haven’t tried the “marketplace” yet…

  • an acute observer

    because nobody knows shit in this world

    What’s worse,

    farming out to the lowest bidder? heh, that’s just another business principle of BUY LOW SELL HIGH
    which profit-driven businesses will always face management problems + end-user customer support

    guess why….. because of NOT MONEY, BUT GREED.

  • Michael Richey

    ‘If there’s a positive message that we can glean from the collapse of the portal, it is that there are a LOT of people interested in getting healthcare via the government.’

    Or they’re hoping to become compliant before the IRS takes up its responsibility of enforcement.

  • Frank Grimez

    “rather than farming it out to the lowest bidder”

    Please back up this statement with facts.

    What you can learn about your government is they don’t have to a budget, especially with healthcare.

  • Noitalever

    The real problem is that whenever anyone sees the government involved they charge 300 times the rate they would charge anyone else. If people would just have a little integrity in this area, our costs would go down astronomically.

    • azt24

      Contractors who have experienced the government procurement and requirements process learn to charge more, or they go broke.

  • Darth_Barney

    Does anyone know what is the underlying framework?

    • NimitySSJ

      Jekyll. It and the companies involved are in the link I posted above.

      • Alex Howard

        I wrote the feature in question. Jekyll is serving text and images, not handling the registration & authentication engine, where the issues are occurring. The companies involved in building the front end are not the ones responsible for the issues; that would be CGI & a dozen other contractors.

  • tricorn

    I’ve seen zero problems with most of the website, it’s clear and very responsive. You can get all sorts of data, filter and sort, download, e.g. the list of all plans available through the exchanges, with representative costs.

    The only part that’s having an issue is the registration and other areas where you’re working with your own account info, i.e. the part where they need to keep it on secure controlled hardware.

    I suspect that the non-individualized parts are on scalable commercial servers, so they have no problem handling the initial crush, then cutting it back. Providing enough capacity on the secure servers would be an unjustified expense after the first few weeks, can you imagine the uproar over how much excess capacity they’d have to have if they could handle the first few weeks with no delays?

    People keep comparing this to Amazon and other commercial sites, that have been up for many years, and have a steady predictable traffic pattern. They’ve also had plenty of their own problems through the years, especially when handling unusually large crushes.

    • scarymatt

      I have no idea what sort of plans are available (aside from the generic descriptions). That’s behind the apply / login stuff, and I’ve never been able to get past it. I think I have an account set up (username, password and security questions), but I’ve never successfully logged in, AFAIK. Maybe I need to try in the middle of the night or something.

      I like how you buried the lede about the problems you have seen after reporting “zero problems.” I don’t understand why we aren’t able to browse without being logged in (or alternatively, why the ability to do so is so well hidden, because I haven’t figured out how to do that). How can the publicly available list of plans be something that needs to be secured? The only thing I can think of is that they don’t want you to see what chumps in other states can get, which is a terrible reason.

      • tricorn

        Go to the bottom of the front page. There are links to available health and dental plans (links to data.healthcare.gov, “Plan information for individuals and families”), with a database browser with filters, sorting options, etc. Filter on your state and county, then sort by plan cost, for example. Or download the whole thing.

        The interface doesn’t work very well with touch, but it’s quite responsive.

        I point out that the web site itself is handling the load just fine, because that part of it doesn’t need to be super-secure, can be scalable to handle the initial load, then dropped back after that; the part that people are having problems with will be fine in a couple weeks, and isn’t handling the load right now because it can’t be farmed out to less secure (but scalable) systems. Overbuilding it to handle the first few weeks would have been a waste of resources.

        • scarymatt

          Thanks! I got suckered into clicking the big “Start Here” button next to the big letters saying, “WANT TO LEARN MORE FIRST?”

          UX? We don’t need to stinking UX!

        • Alex Howard

          Spot on. It’s the secure authentication (account creation / security question) that depends on queries to SSA & DHS that people are reporting issues with everywhere. The website is scalable on Akamai, the engine is not, which is why I gather they’re physically adding servers. Sharp observation on the data being available.

          • scarymatt

            Yes, but why does the observation have to be so sharp? Consider, all I wanted to do was browse what was available. I’ll bet a lot of people wanted that, too. But because you guys hid THE MOST IMPORTANT INFORMATION, everyone thinks they have to log in! WTF!

            You guys failed, too.

    • ShimC

      Are you serious? Are you talking about the Federal Exchange? Not a state exchange that the FFE passes you off to? I tried several times every day and wasn’t able to even create an account until yesterday. and then nothing else worked for me. 25 years in IT and I’m pretty sure “it’s not me.” To me, the FFE/FFM is an abject failure. Total debacle so far. That information you note at bottom of the page (Search & Browse Datasets and Views) is too difficult/meaningless and of no value to ordinary people.

      • tricorn

        I’m talking about the web site itself. It’s working fine and handling the load with no problem. That’s the part that doesn’t need to be secure. The secure part is where all the problems are.

        I was able to create an account the evening of the 2nd. It did have a few glitches – the first time, it got an error and I had to start over. The 2nd time, after confirming with the link sent via e-mail, the next page gave me a “Access Forbidden” page. Then, as with many people, I didn’t get a login page, just the busy page.

        Today I finally saw the login page, but there’s now a glitch where it won’t recognize your account name/password properly. I can click on “I forgot my password”, and it sends a link to my e-mail address (so, the user name is correct and associated properly), but clicking the link gets “We weren’t able to process your request because we couldn’t find a Marketplace profile that matched the information that you provided.”

        I called the phone number, after a 25-minute wait (pretty close to the estimated 20-minutes given at the beginning) I spoke briefly with someone who confirmed that many people are having the same problem, there’s nothing wrong with my account, and I should try again sometime later. I feel sorry for them, they’ve trained hard to learn all about insurance coverage and they’re answering questions about login failures.

        This isn’t anything that hasn’t happened on big rollouts with many companies, e.g. World of Warcraft, the GTA release just a few days ago, Apple, many others.

        I fully expect to be able to log in within a few days. Given that the earliest deadline for signing up is Dec 15, I have no problem with that.

        I mentioned the dataset browsing in response to the comment about that information not being available until you logged in. I agree that it isn’t really appropriate for most people, and it’s also not really obvious that it’s there, but it is available.

        • Incompetence

          Based on your first sentence: “I’ve seen zero problems with most of the website, it’s clear and very responsive” followed by your subsequent message: “It did have a few glitches – the first time, it got an error and I had to start over. The 2nd time, after confirming with the link sent via e-mail, the next page gave me a “Access Forbidden” page. Then, as with many people, I didn’t get a login page, just the busy page”, you have lost all credibility as a judge of functionality.

          • tricorn

            “most of the website”

            Obviously, I haven’t been able to get to parts that require you to be logged in, and obviously those are some of the most important parts (indeed, without those parts, anything else is of no consequence).

            I wasn’t judging anything on functionality, I was talking about the Web Site Servers – they work, they work fine, they show no problems with the load.

            All of the issues are with the back end. Some of them are load related, those issues have mostly gone away.

            There are some additional problems, and those need to be addressed. Whether my specific issues are caused by my account being glitched because I created it during a heavy load time or there’s something else going on, I don’t know.

    • http://burningbird.net Shelley Powers

      As far as I know, people haven’t been able to get to the plans. You can’t get to the plans until you register.

    • Garys

      You forgot to mention that you spend most of your time stoned.

  • sdinfoserv

    The IRS tried for 6 or 7 years to build an internal appliation that eventually ran billions over budget. Utlimately is was such a failure it was outsouced to EDS.
    As a Govie, I can say the reason this happens is because of frequent change in leadership. Priorities, directions and even management changes result in dynamic and sometimes dramatic changes in scope. As every developer knows, that’s the kiss of death for any project from the beginning.

  • NimitySSJ

    This might be enlightening for you people. The development wasn’t done to the lowest bidder far as the link says. It was given to people who tried a very radical approach. And maybe effed it up. ;)

    • http://www.mixologic.com Ryan Aslett

      There was nothing radical at all about their approach. It was proven and tested during the Obama Campaign, and is actually an older, more venerable approach. (Static HTML sites).

      The important thing to realize here is that when building out a massive government project like this, in the timeframe they were given (probably nowhere near enough), with the sheer number of stakeholders involved and the massive number of requirements and laws, you are really not looking at a project that even remotely resembles a private company product buildout.

      • CoderForHire

        After three years and $313 million, it is still utter crap. With that amount of time and money, I could, by myself, create a website and deploy it without the kind of fail this site has.

        • nitelite_ew

          To be fair, if you had been alone on that project you wouldn’t have had time to write a single line of code in those three years, cause all your time would have disappeared in meetings updating stakeholders on your lack of progress, discussing changing requirements and working with the lawyers to map out all the legal requirements in the different states.

      • NimitySSJ

        The radical I was referring to was for a big government deployment.
        They often use old tech, waterfall development, etc. This one hired
        smaller design houses, used Jeckyll, etc. Nothing fancy far as private
        sector is concerned but seemed surprising coming from the govt. That’s
        all I was implying.

    • Alex Howard

      As it happens, I wrote that piece. The approach taken to building the front end (as Ryan mentioned below) worked fine. The issue is on the backend. CGI Federal & a dozen other contractors built that code.

      • azt24

        If this front end worked fine, I would hate to see what a failure looks like. As for the issue being on the back end, did you miss the part where James Turner talked about being left in a “deathtrap of UI” from which he could not escape?

  • Rob Lewis

    You say “Unfortunately, that has led to what is effectively a DDoS attack.”

    Why not consider the possibility that there is in fact an organized DDoS attack staged by the anti-Obama crazies? I consider this more likely than not.

    • azt24

      Check out this page of comments from Reddit on the un-optimized source code (sample “They’re loading 11 CSS files and 62 (wat?) JavaScript files on each page, uncompressed and without expires headers.”)


      With code like that, 1000 concurrent users could unwittingly stage an effective DDoS attack.

      • Anarcissie

        That doesn’t obviate the possibility that some people have been doing sabotage.

        • scarymatt

          They’ve already admitted they based their scaling on medicare’s web site. Occam’s razor says their poor planning was enough to make it all suck without any malice. Your tin foil hattery just makes you look like a nut.

          • Anarcissie

            I’ve observed deliberate sabotage of corporate projects. Why not government projects that have incurred a lot of ideological excitement?

          • scarymatt

            If you had any evidence at all, that would be interesting. But I haven’t heard anything other than people making it up, like you. And if, for the sake of argument, there actually was some sort of malicious DDoS going on, it still seems likely that the natural traffic would be enough to kill the site if the DDoS went away.

        • Garys

          As a programmer I know how frustrating poorly functioning code can sometimes be. Initially you think: I wonder if it’s being abused? But after you dig in, it’s always a bug. They simply released this before it was ready.

          • Anarcissie

            I was thinking of internal sabotage. I’ve observed people do it at some large corporations in order to make other people look bad. But then, they say you should never ascribe to malevolence what can be explained by stupidity.

      • http://burningbird.net Shelley Powers

        Agree. No one has to attack a site that’s poorly designed. It attacks itself.

    • Garys

      So if someone disagrees with a politician you like – they’re crazy? It never ceases to amaze me how people who assert they’re “open-minded” and “tolerant” are actually closed-minded bigots.

      • Rob Lewis

        Not at all. But if someone won’t let go of the idea that Obama wasn’t born in the U.S., or is a closet Muslim secretly trying to institute Sharia law, or is planning to enslave us all with—horrors!—health insurance then, yes, that person is crazy.

  • Texan on Cobra

    The reason contractors charge three times as much for government projects is that they have three times as many hoops to jump through. There is so much legislatively-induced oversight that for every productive worker you need at least two or three others – to study the problem and document the need out the wazoo, to monitor and report on progress to the nth degree, and to conduct exhaustive post-mortem reports documenting what was accomplished. Then there’s the lengthy and incredibly convoluted process of getting the work in the first place. It’s a nightmare!

    And it’s not just journalists and politicos checking out the site, but they may well be hashing things up for those is us who are genuinely interested in signing up. I hope the scheduled maintenance this weekend will fix things, and am posted at Congress for the shutdown which is no doubt contributing to IT resource availability for troubleshooting.

  • http://refinch.com Bob Finch

    ” If there’s a positive message that we can glean from the collapse of
    the portal, it is that there are a LOT of people interested in getting
    healthcare via the government.”

    Or, it might mean that there are a lot of us hitting up the site to laugh at and mock it… DDOS by proxy, so to speak.

    I have been there and done that with a large government IT project that ended up never launching. And that was just an internal site for statewide accounting. This project is doomed.

  • http://ducknetweb.blogspot.com/ Medicalquack

    You can compare it to the California State payroll system, over 300 million and 20 years and nothing yet, now suing SAP the last contractor and now the old COBOL system is still doing the job, fat code, yes. IRS, Homeland Security, Social Security all that the federal data hub needs to work with, COBOL and middle ware needed for scaling no doubt about it.

    I said in the beginning on a post a few months ago, “will it blend”….this is a big project and yes HHS should have called it a “beta” and didn’t get started early enough. I discussed all of that back in the beginning of August and tried to put it out there realistically.


    Sad truth that the public and government are all learning is “the short order computer code kitchen burned down years ago”..in other words it’s a lot of time and effort and you can’t speed it up as programmers are already at full speed. Those folks are getting a beating for sure and it’s not their fault. We have a total digital illit running HHS but that’s another story and her actions have been “choice” no less stumbling all over herself along with other politicians out there trying to inform the public:) Just be honest is the best policy.


  • Chipmonkey75

    “The biggest takeaway though, is that the way that the federal government bids out software is fundamentally broken.” — Government software contracting IS fundamentally broken, but the issue goes far beyond the “lowest bidder” issue. I’ve worked with many government institutions on software development and they each have very specific requirements around competing the bids, but also around the development itself. Outdated software development techniques are MANDATED in some places (I’m primarily familiar with DoD and State governments, but it wouldn’t surprise me here). Requirements go through mountains of overhead, developer creativity is often stifled by red-tape. Privacy and security are controlled by much tighter laws and policies in government than in corporations (HIPAA has nothing on the PII hoops in the DoD).

    There are groups that can get around this red tape… some government technologies break the mold for the better. But it’s not nearly as simple as hiring Amazon or Intuit and leveraging their expertise. It SHOULD be that simple, and these problems need to be fixed, but your last paragraph feels like it lays more blame at the feet of the lowest bidder than may be deserved. I’m just pointing out that it’s far more complicated than that (even if it shouldn’t be).

  • http://burningbird.net Shelley Powers

    And you have to be consistent.

    When we could create our accounts before the marketplace opened, there were no constraints on usernames. Now I have a username that is technically invalid. So far it seemed to work, except today. But I don’t know if the problem is because of my username, or because the system is (as usual) not working.

    Of course, when the leader behind all of this uses words like “geekalicious”, what can we expect?

  • http://burningbird.net Shelley Powers

    PS Talked to customer service, and she said everything is not working today. So much for the weekend fix.

    I really wanted to know what my premiums would be, but I’m giving up. I’ll have to try in November. I can’t keep wasting my time.

  • dave riley

    After you finish the application you need to click the green square with the word “SET” in it. This eluded me for a long time as there is no indication of it being a link. No mouse-hand-pointer or color change when you scroll over the green “button”. Is like something a little kid would make in high school.

  • Man_On_Fire

    Obama really doesn’t care if the site actually works or not. He is just interested in having his law passed and the government taking over a huge chunk of the private sector. As with anything the government does, results don’t matter and the fix always requires more government control.

    • thirstydc

      Remember that the health exchange is for private insurance. There is no public option a la most of Western civilization.

  • Jim Shaffer

    If the government did something intelligent like buying much of the product “off the shelf”, beltway bandits would literally tie up the process in appeals and lawsuits until the product is no longer needed. It’s tough working with government.

  • David

    When Dick Cheney was asked why they hired Halliburton (in no-bid contracts) to do the rebuilding in Iraq, he answered something to the effect that it was the only company that could do the job at that scale.

    • Garys

      Perhaps that was true. Although, I think the next question is: should we even be doing something on such a scale???

  • Kelly Anderson

    One reason the government can’t just hire the best is that the best often have an anti-government stance. High intelligence doesn’t always lead to being a fan of the government.

  • Scott Riley

    ‘If there’s a positive message that we can glean from the collapse of the portal, it is that there are a LOT of people interested in getting healthcare via the government.’

    Yes, it’s always a good thing when people seek to use the government to live off others.

  • John Incognito

    635 million dollars later, there’s quite a few software executives that have lined their pockets with taxpayer money while delivering a substandard product at best. Our government in action.

  • Richard Stewart

    Professor Dijkstra once said that sometimes, before embarking on a project, one should try very hard to kill it, stop it, first.

    That is the elephant in the room which the polticial backers of the PPACA do not want to discuss; now that elephant has just defecated on the carpet, spreading filth and making him even harder to ignore. But the politicians will work even harder, holding their noses and giving their friends fancy deodorizers as the mess grows larger…

  • Randy Long

    There are three attributes of a project, good, fast and cheap. Improving one requires reduction of the other two. If a project needs to be completed fast and good, it must be accepted that it will be expensive.
    As the project expanded to include setting up exchanges for most states, good, fast and cheap needed to be addressed. Ideally, the budget would be expanded to cover the additional work. This was not done, and the deadline could not be adjusted. Consequently, the project teams needed to work longer hours and take shortcuts. Basically, the result should have been expected.

  • Rick Kasten

    You ask “Why can’t the government draw on this kind of expertise when designing a
    site as critical to the public as healthcare.gov, rather than farming it out to the lowest bidder?”

    Easy. Because the government employee charged with the responsibility of judging the scope of the project, and thereby the expertise of the organizations responding to the RFC, is clearly unqualified to perform either of these functions. The “government” cannot draw on this expertise, because they would not recognize it as expertise if it was delivered to their doorstep attached to a flashing neon sign that read “THIS IS EXPERTISE”. This fact of the government’s incompetence should come as absolutely no surprise in the slightest.

  • john

    Wasn’t put out to lowest bidder: worse it was sole-sourced so no competition

  • Harihara Vinayakaram

    It shows that Oreilly does not practice what it preaches :-) . I mention this because this post is mentioned as a popular post in the programming.oreilly.com portal but the link does not work since the first item does not contain a http and is just a string . The other links are fine .