Dither and Bicker

I loved the healthcare.gov debacle. Loved it. Not getting into politics here, just looking at it objectively as a large scale, publicly-facing, web application. Web development does not typically become a topic of national discourse, especially in my part of the world. But for a brief window in late 2013 the man on the street was at least aware of my profession. They even had opinions on tech and wielded newfound terminology in their arguments. "The government shoulda went with open source" "They shoulda worked with Google" and on and on, a wonderful time.

Healthcare.gov was a pretty serious application to serve as an introduction to the industry. Out of the gate it needed to juggle a multitude of APIs and a crush of traffic. As the initial bugs became headlines, I had to chuckle to myself. These were pretty common failures of a new site. I’d encountered (and introduced) similar problems in various projects during my career.

Around the same time my team was again hiring for new talent, working on perfecting the “script” of questions for interviews. It struck me that since the problems encountered by healthcare.gov were somewhat common to enterprise applications, they’d make for great questions for a perspective new hire.

Unfortunately, we hired someone before I could put any of this into practice, but here’s potential questions. These are more questions for discussion, nothing requiring code examples. I’ve added the kind of answer I’d be fishing for from an applicant to most of these. Maybe on the next hiring go-round I'll get to try some of these out.

Deep Dive Question 1

Your web app needs to get information about the user’s family. A series of forms collects individual siblings, children, spouse, etc… Users are reporting that duplicate family members are showing up.

What could allow data duplication on the backend?

Assuming a relational data store, this could be due to a lack of unique keys.

What are some possible causes on the frontend?

The form could take a long time to complete, during which the user is presented with a loading prompt that spins indefinitely, or worse the page doesn’t have such a cue and the user is left wondering if they’ve submitted anything at all.

It's perfectly acceptable to push “refresh” if you think the page has stalled, and the web app needs to account for it.

Deep Dive Quesion 2

Assuming you had control just over the backend, what steps could you take to prevent this from happening?

Once again assuming a relational data store, you could make a multi-column unique key. Social Security numbers are fairly unique, so they’d be a natural fit.

For Follow-Up:

Throw out the assumption that the family information needs to be complete before being saved in the application. Say that a user only needs to fill out the name and relationship of a family member, but are allowed to enter their social security number at a later date. Also assume that there can be typos in all these fields, will your original assumption hold?

Deep Dive Quesion 3

You only control the frontend, backend’s balked and said they can’t add unique keys because it would impact performance or a similar lame excuse for sitting on their hands. What could you do to prevent duplicate submissions just on the front end?

Few strategies here: first make sure than a user can’t submit a form multiple times. Deactivate any submission buttons in addition to presenting on screen processing indicators (aka a spinner). That’s the easy part.

A more thorough solution would be to enter all family members on a single dynamic form that would be submitted all at once.

Solving this issue on the backend exclusively or the frontend exclusively still leaves gaps, what are some of the issues?

You don’t really solve the case of refreshing the page. Database errors are pretty graceless, etc...

For Follow Up:

Assume you could make changes to both the backend and the front end, what could you try?

In addition to the frontend and backend specific options, you could also generate a request number to associate with each potential form request. It would be generated when the form was rendered. Therefor any resubmissions would contain the request number. The backend would keep track of the request numbers, and if an existing one was found it could ignore the new form data altogether.

Additionally, if the front end sent all of the family information at once, you could destroy and recreate the family information on the backend with every request.

Many options here...

Deep Dive Question 4

Your system will leverage 55 existing APIs to build a list of products presented to the user. It’s a safe assumption that at least one and at most 10 of the total 55 will be consulted at a time, as determined by factors such as a user’s state of residence or military background.

What are some of the challenges with dealing with APIs?

First off, every API is unique, even if they all support a common spec. More than likely each will use its own means of communication. Some may be RESTful JSON, or RESTful XML, or SOAP. Otherwise it’s the wild wild west. Some will be “RESTful” post url encoded plain-text. Others may be the older, ghastly, API variants. The type where “first fifteen characters are the command, the remaining fifty are the value for the command”, or “the first two bytes describe the length of the message”.

You also can’t assume that every API will be equally responsive. Some respond to a request in under a second, some may take longer.

How would you approach handling all these disparate APIs in your code? Bear in mind you’d be performing a similar operations for all of these.

Here I’d really hope to hear something about generic interface for all the service operations. Getting a plan from API 1 and API 20 should look the same.

For Follow-Up: What are some of the difficulties in making a generic API?

If there’s no set standard for requesting data from an API, each API may need a different set of steps to provide the desired result.

Add to this a need to authenticate, which may require multiple requests (or handshakes, colloquially) depending on the type of authentication. All of this would need to be hidden behind a standard interface in order to issue a simple command.

Deep Dive Question 5

There’s a separate API that is used to determined subsidies and savings. Every enrollee is eligible for subsidies depending on a mix of income, family size, and military service. The problem is while your site can handled 10,000 concurrent connections, this particular API is pretty slow and can only handle 100 concurrent connections. Even after getting a connection, it can take more than a minute to retrieve a response. Every applicant that can qualify for subsidies needs to be vetted by this API. How would you handle this?

Long question with a short answer. I'd want to here about "batching" or "queuing" requests.