November 24, 2013

This is the post where I paraphrase 10 things in the NYT article "Tension and Flaws Before Health Website Crash."

Here's the text, by Eric Lipson, Ian Austen, and Sharon LaFraniere. I'm blocking and indenting their text and boldfacing key words that made me feel compelled to paraphrase so I could see what they were muting or failing to pursue with investigative vigor.

1. Government officials and its contractors were in conflict, and some people — who? — made questionable decisions and demonstrated poor leadership.
[T]ensions between the government and its contractors, questionable decisions and weak leadership within the Medicare agency turned the rollout of the president’s signature program into a major humiliation.
2.The Obama administration, dazzled by its grandiose idea of making a dazzling website, refused even to engage with the reality that was plaguing the computer technicians: It was impossible to meet the deadline with a website that even worked.

The prime contractor, CGI Federal, had long before concluded that the administration was blindly enamored of an unrealistic goal: creating a cutting-edge website that would use the latest technologies to dazzle consumers with its many features. Knowing how long it would take to complete and test the software, the company’s officials and other vendors believed that it was impossible to open a fully functioning exchange on Oct. 1.
3. Delusional Obama officials panicked and only interfered in ways that made the impossible task even more difficult.
CGI and other contractors complained of endlessly shifting requirements and a government decision-making process so cumbersome that it took weeks to resolve elementary questions, such as determining whether users should be required to provide Social Security numbers. Some CGI software engineers ultimately walked out, saying it was impossible to produce good work under such conditions.
4. The truth is being suppressed. What would we hear from a full-on whistleblower?
“Cut corners, make date,” said one specialist, who like most of the people interviewed for this article would not allow his name to be used because the Obama administration has requested that all government officials and contractors involved keep their work confidential.
5. What the hell is MarkLogic and why did it get this sweet deal that caused so much grief?
Another sore point was the Medicare agency’s decision to use database software, from a company called MarkLogic, that managed the data differently from systems by companies like IBM, Microsoft and Oracle. CGI officials argued that it would slow work because it was too unfamiliar.
6. There was a competent person who could have been in charge, but he was identified with Romneycare.
The Medicare agency was not everyone’s first choice to run the $630 million project. White House officials at first debated whether to name an outsider, such as Jon Kingsdale, who set up the landmark Massachusetts health insurance program, or even to create a new agency.
7. The one named informant is Wallace Fung, who seems sympathetic to Henry Chao, the completely unqualified/unempowered administration official who got stuck in charge of the project, and Fung tells us that Chao was freaked out about the delusional, disastrous demands coming from the White House.
As a result, the president’s signature initiative was effectively left under the day-to-day management of Henry Chao, a 19-year veteran of the Medicare agency with little clout and little formal background in computer science. Mr. Chao had to consult with senior department officials and the White House, and was unable to make many decisions on his own. “Nothing was decided without a conversation there,” said one agency official involved in the project, referring to the constant White House demands for oversight....

One evening last summer, [Chao] called Wallace Fung, who retired in 2008 as the Medicare agency’s chief technology officer. Mr. Fung said in an interview that he told Mr. Chao to greatly simplify the site’s functions. “Henry, this is not going to work. You cannot build this kind of system overnight,” Mr. Fung said he told him. “I know,” Mr. Chao answered, according to Mr. Fung. “But I cannot talk them out of it.”
8. The original plan was to build the sites for the individual states that declined — as was their option in our constitutional system — to build websites, but somewhere along the way, for some reason — political? — the White House decided it wanted one big federal website, and this switcheroo wreaked havoc on the design work.
A pattern of ever-shifting requirements persisted throughout the project, including the administration’s decision late last year to try to redesign the site’s appearance and content to make it more informative to consumers, according to many specialists involved. The administration also decided to reconfigure it as a national site, instead of one where each state had its own front page, after many states decided not to open their own exchanges.... “It was monstrous, a monstrous impact,” said one specialist about the amount of code that had to be rewritten because of the redesign and other similar changes.
9. Administration officials trying to cover their asses got paranoid that the technicians were trying to cover their asses. (Maybe, like me, they didn't know whether using code to patch a flaw is a coverup or actually the way you fix flaws in code.) And like a bunch of social workers they reinterpreted the technical problems as personality conflicts and wasted the technicians' time making them drive to Baltimore to participate in some inane make-nice therapy group. (Oh, how I wish I could see the actual quotes from the unnamed CGI and QSSI people who, I suspect, were royally pissed at this idiotic exercise!)
Eventually, Medicare agency officials began to suspect that staff members at CGI were intentionally trying to hide flaws in the system, to cover up for their inability to meet production deadlines. They ordered CGI technicians to drive from their offices near Dulles International Airport in Virginia to the agency headquarters near Baltimore to review their code with government supervisors. The Medicare agency was also growing frustrated with tension among contractors, noting that initial tests of parts of the system were being delayed because of “coordination issues” between CGI and QSSI, which won another part of the job after losing the lead contractor role.
10. Obama lied straight-faced at us or his people completely tricked him into talking up the website and how great it was going to be. When he could have used his charm to manage and mellow our expectations, he inexplicably chose to amp us up.
Despite the behind-the-scenes crisis, the president expressed confidence about the exchange just days before its debut. “This is real simple,” Mr. Obama said, during a speech in Maryland on Sept. 26. “It’s a website where you can compare and purchase affordable health insurance plans side by side the same way you shop for a plane ticket on Kayak, same way you shop for a TV on Amazon. You just go on, and you start looking, and here are all the options.”

87 comments:

Bob Ellison said...

This is not "technology". This is database and network stuff. We sent a man to the moon, and these bozos can't even balance a checkbook. This is very basic computer stuff that was mastered forty years ago.

Eric Jablow said...

MarkLogic is a well-respected company that specialized in building databases for companies to store documents and set them us for searching. However, they create "NoSQL" databases, which is a common popular trend these days, and these compete with older standard relations database that IBM, Oracle, Microsoft, and others put out. Integrating with those companies' offering will be difficult.

Eric Jablow said...

Oops--"Relational", not "Relations".

khesanh0802 said...

I think it's amusing that the NYT is just getting around to this reporting when so many other papers had this story a month ago. Is this a CYA moment? What's next?

BTW I love Ann's comments.

Lance said...

This is very basic computer stuff that was mastered forty years ago.

I'm sure that's what the Obama admin thought. But it's not true. Web development is reinventing itself about every three years.

What the hell is MarkLogic and why did it get this sweet deal that caused so much grief?

MarkLogic is a non-relational, noSQL database. It's used for large-scale document data. Amazon, Facebook, etc. have all switched to these types of databases (but not that particular one), specifically because they're so much faster under heavy load.

I agree it's surprising that CMS would insist on a technology that their lead contractor didn't like. But I'm actually more surprised that a supposed big hitter like CGI Federal wouldn't know how to work with a noSQL database. MarkLogic has a Java interface, if CGI were staying current on their technology it shouldn't have been that big of a jump for them.

Lance said...

@Eric Jablow
Integrating with those companies' offering will be difficult.

Integrating a legacy system with a new db technology can be challenging, but companies do it all the time.

In this case, though, CGI is writing a NEW system. I don't think the learning curve for MarkLogic was as big a concern as the NYT article made out. I think the big issue here is the aggressive deadline combined with the poor requirements management.

AJ Lynch said...

Nancy Deparle was one of the Obama admin architects / mahoffs who designed this abortion of a law and system. Her husband is Jason Deparle who, I believe, is a reporter for the NYT. The NYT should disclose this in every Obamacare story IMHO.

Sorun said...

"a cutting-edge website that would use the latest technologies to dazzle consumers with its many features."

Obama Admin: "We want stuff like those Google doodles. Those are so cool."

Michael K said...

Maybe he answer is here.

In the early days of print, you had to understand the tech to run the organization. (Ben Franklin, the man who made America a media hothouse, called himself Printer.) But in the 19th century, the printing press became domesticated. Printers were no longer senior figures — they became blue-collar workers. And the executive suite no longer interacted with them much, except during contract negotiations.

This might have been nothing more than a previously hard job becoming easier, Hallelujah. But most print companies took it further. Talking to the people who understood the technology became demeaning, something to be avoided. Information was to move from management to workers, not vice-versa (a pattern that later came to other kinds of media businesses as well.)


This is so characteristic of this bunch that it is obvious.

A Salient Voice said...

All pretty apparent to (and foreseen by) outside observers with software experience. http://www.salientvoice.com/journal/2013/11/3/the-us-government-is-developing-a-useful-software-engineerin.html

cubanbob said...

Even if the website and the databases worked perfectly it would make no difference, ObamaCare is fundamentally a disaster. Indeed if the site and databases worked perfectly it would accelerate the demise of ObamaCare as millions of people would see in near realtime just how bad it is.


Jack Wayne said...

And now all we have to do it wait for the other show to drop. Since Chao has acknowledged that the back end payment part is not even written, how long until that is done. My guess is another 3 years. In the meantime how are the insurance companies supposed to do business on these policies when they are not getting paid by the government?

Get out the popcorn. Obama will be in the single digits by 2016.

betamax3000 said...

My 3-item list:

1. Incompetence

2. Corruption.

3. Hubris.

You can make the appropriate Venn diagram with Obamacare in the middle.

Ann Althouse said...

"Obama donor’s firm hired to fix Web mess it created." That's about QSSI.

Meade said...

daz·zle verb \ˈda-zəl\
of a bright light : to cause (someone) to be unable to see for a short time

: to greatly impress or surprise (someone) by being very attractive or exciting

betamax3000 said...

RE: “It’s a website where you can compare and purchase affordable health insurance plans side by side the same way you shop for a plane ticket on Kayak, same way you shop for a TV on Amazon. You just go on, and you start looking, and here are all the options.”

Should have given the project to Amazon in the first place. Then we could buy health insurance through the Althouse portal.

Ann Althouse said...

Give 'em the old razzle dazzle
Razzle Dazzle 'em
Give 'em an act with lots of flash in it
And the reaction will be passionate
Give 'em the old hocus pocus
Bead and feather 'em
How can they see with sequins in their eyes?
What if your hinges all are rusting?
What if, in fact, you're just disgusting?
Razzle dazzle 'em
And they'll never catch wise!


That's from — so apt! — Chicago.

YoungHegelian said...

Nice point by point breakdown, Professor.

But is there anyone who wasn't an Obama acolyte who's surprised by this? That Obama's politicos were incompetent micro-managers has been an open secret around DC since day 1.

The only surprise is that this perhaps fatal wound has been completely & utterly self-inflicted, when all along the way the Obama administration could have traded time for considerable political advantage from the Republicans. The multiple failures of management of the healthcare.gov project show an administration that is pathological from top to bottom.

God help us all if a major political crisis (e.g. China attacks Taiwan or Japan) happens on these guys watch.

Paul S said...

This blog keeps getting better and better. Great post Ann.

Ann Althouse said...

"Should have given the project to Amazon in the first place. Then we could buy health insurance through the Althouse portal."

Thanks. And please do tell me you enjoyed this post by shopping — now or soon — through the Althouse portal, but there's a big difference between Amazon and what the govt needed, which was to force everyone to buy and to buy the same thing. Maybe there were different sellers and different prices, but you must buy insurance, and you can't get the cheap kind anymore. The government has already decided what you should like.

YoungHegelian said...

Oh, and by the way, do any of you think for a New York minute that all the emergency measures & staff the Administration is calling upon to fix healthcare.gov are legal under federal contracting law?

Political embarrassment does not constitute a national emergency. If the Republicans take the Senate & hold the House, expect hearings on this topic.

betamax3000 said...

All hoist and no petard.

Ann Althouse said...

"I agree it's surprising that CMS would insist on a technology that their lead contractor didn't like."

They didn't just not like it. They were not familiar with it!

Meade said...

"Then we could buy health insurance through the Althouse portal."

It's coming. Soon as we get the bugs and glitches tweaked out. Meanwhile, feel free to purchase "Health Insurance for Dazzled Dummies". $14.14 Paperback

Terry said...

I haven't read the Times article (hit my limit for this month), but I'm puzzled by all the references to problems with coding and integrating incompatible systems.
From what I've read elsewhere, the big problems that are causing delays are administrative and legal. A lot of individual tax payer and health information has to be shared between agencies and there are protocols and safeguards that get in the way of this.
Meetings must be held, legal opinions rendered, etc.

Meade said...

Also, feel free not to purchase "Health Insurance for Dazzled Dummies". $14.14 Paperback.

No pressure. No coercion. No mandate. It 's the Althouse way.

Ann Althouse said...

"CGI officials argued that [MarkLogic] would slow work because it was too unfamiliar."

I'm going to paraphrase that: CGI officials told administration officials didn't know how to use MarkLogic, but they were told they had to use it anyway.

Imagine getting hired to do a job you believed you knew how to do and showing up at the job site and finding a lot of tools completely unlike the tools you'd used on all the other jobs you'd done, that were the basis of your expertise. This is the stuff of nightmares!

Skeptical Voter said...

Obama approval in single digits by 2016 may be correct--but as the Brits say, "That butters no parsnips." Unless Obozo decides that the constitutional two term limit doesn't apply to his wonderfulness, he's out the door in 2016.

And his wrecking ball is going to keep on rolling until at least after the 2014 Congressional elections.

Ann Althouse said...

Imagine being a surgeon, the only surgeon on call, required to save a life, and you go into the operating room and there are all the tools laid out on the tray, and they're completely different from the tools you've used before.

This made me think of the scene in "Sleeper" where Woody Allen and Diane Keaton are suddenly in an operating room where they must — before an audience they must bamboozle — reconstruct the dead leader using the one living part of him that has been "kept alive": his nose.

Ann Althouse said...

"I haven't read the Times article (hit my limit for this month)…"

Your limit won't stop you when you enter from a link (like mine).

It will count against your total if you haven't hit your limit, but if you use outside links to go in, you can always go in.

That's why I subscribe to the Times, so I can always blog whatever find.

And why I ended my subscription to The Wall Street Journal, which doesn't do it that way.

pm317 said...

The administration also decided to reconfigure it as a national site, instead of one where each state had its own front page, after many states decided not to open their own exchanges.... “It was monstrous, a monstrous impact,” said one specialist about the amount of code that had to be rewritten because of the redesign and other similar changes.

This is the key fuck up. Distributed versus Centralized.. classic fuck up. Looks like politically motivated because if every state had its own front page and went to its own database, how could people distinguish it from other state exchanges that built their own? Obama wanted the big Federal government stamp on the front end and back for the states that didn't want to build their own exchanges. I don't understand -- so were they bringing all 32 states data under one database and trying to route user's request searching one humongous database? Does not make sense, though it explains why it was crashing.

pm317 said...

No obvious political links to Marklogic but as others mentioned, nosql makes sense for big unstructured data and is falshier and newer. However, how unstructured is this application? It is not as diverse as Amazon and Amazon does it because of analytics. Where is the need for analytics for this government application, unless they are trying to harvest people's data/activities for something else.

Martin L. Shoemaker said...

"Imagine getting hired to do a job you believed you knew how to do and showing up at the job site and finding a lot of tools completely unlike the tools you'd used on all the other jobs you'd done, that were the basis of your expertise. This is the stuff of nightmares!"

Not that I want to defend this mess IN ANY WAY, but... welcome to my world. Technology choices are often made by whatever the CEO read about in some magazine lately, by whoever lobbied for business and paid the largest kickbacks, or by folklore and trends. The programmers almost never have a choice (though we may stealthily supplement the tools we can't use with those we can); and it's not unusual for our managers to have no choice, either.

Healthcare.goof is a software engineering failure of epic scale and visibility, and I hope a lot of lessons are learned. But at its core, it's following the patterns of thousands of other failed projects. When you start with the deadline first, refuse to define the requirements adequately, grow and change the requirements right up to release, and keep commanding "Failure is not an option," THIS IS WHAT HAPPENS. EVERY. SINGLE. TIME.

Martin L. Shoemaker said...
This comment has been removed by the author.
Michael K said...

This bunch is totally political. The rest is Cargo Cult. They don't know what they don't know. Maybe someone should buy Rumsfeld's book for them.

Hagar said...

Young Hegelian,
What I look for to happen is that Iran demonstrates it has nuclear delivery capability, then annexes Iraq and dares the world to do anything about it.

And then???

Left Bank of the Charles said...

On a project like this, you can manage on staffing, scope, schedule, and quality. Adding staff has diminishing and, past a point, negative returns. With a creeping scope and an unbending schedule, the only thing the line developers have to manage us quality. And so they will sacrifice quality in this situation, every time.

What they needed was more time on the schedule, or a reduced scope. Why didn't they get it? The administration decided, even if that decision was by default, that it would rather have a big flop now, rather than a flop in 6 to 12 months, even if that flop might possibly be much smaller.

That might even not be the wrong decision, as the feedback from the public is qualitatively better. Effectively they have been running a pilot program, which is what they should have done.

The Drill SGT said...

Lance, Though the whole project was FUBAR from the start, your analogy misses the difficult technical challenges. Amazon or Google don’t have to interface with a bunch or legacy COBOL batch systems that were never designed for interactive remote access. I suspect, but don’t know that many of the time-out issues are the result of those database fetches against legacy apps.

IMHO they should have left the legacy apps alone and built a datawarehouse in the CMS facility that would be updated nightly from the systems of record. Then the Obamacare web site would have much faster and consistent response…


pm317 said...
This comment has been removed by the author.
pm317 said...

Then there is this

The Drill SGT said...

The prime contractor, CGI Federal,

That obfuscates a bit. CGI had a prime contract, that is true, and they were the major contractor, but they weren't the "system integrator" (SI). CMS took that role. Meaning, CGI was paid to do parts, and QSSI, other parts, and CMS was coordinating the failure...

The Drill SGT said...

Left Bank :)

scope, schedule, and quality.

Pick TWO

YoungHegelian said...

You know, there's enough technical & managerial know-how on this thread that maybe we should all get together & put in an unsolicited (to say the least!) proposal to come in and fix up that gol-durned, pesky web site for 'em. I mean, could we do any worse?

We can call our newly minted company The Althouse Collective, Ltd.

Glenn Howes said...

Well obviously code is used to patch flaws. But the problem here was likely whether using hacky code as a shim to repurpose already written code when a higher level redesign was called for. Every software engineer reaches that decision point sometime in a project, and it's a test of character for both the engineer and whomever is managing the project.

The Drill SGT said...

Althouse, this quote from the article, backs up my SI post and figures into your #9

Agency officials complained that CGI missed crucial deadlines and that it could not control other contractors, although the company said it had no power to do so.

MadisonMan said...

I think the big issue here is the aggressive deadline combined with the poor requirements management.

Yes. Feature Creep is a bitch when you're developing something. I hate hate hate it, and the stuff I've built is tiny compared to the health care boondoggle. Hitting a moving target is nigh on impossible.

I am not persuaded by the argument that the chief contractors were unfamiliar with a software product, and that that caused problems of the scale that exist. Software engineers do incorporate foreign -- to them -- packages with amazing frequency. It's not hard to do.

The Drill SGT said...

Sixteen companies were prequalified to bid on the project, according to administration officials. CGI was picked as the prime contractor over three other bidders: IBM, QSSI and Computer Sciences Corporation. But the Medicare agency reserved the role of general contractor, or system integrator, for itself, even though it lacked the necessary in-house software engineering resources to handle such a task.

Hagar said...

And, of course, YH, if they take Iraq, they can't very well stop there; they are also going to need Kuwait and the Emirates, and ....

pm317 said...

More on why MarkLogic..and why this government application is not like Amazon.. Amazon wants to for example analyze millions of search strings (which are unstructured and unpredictable) and therefore might want to use something like MarkLogic.. What is so unpredictable about getting your income, preexisting conditions, number of children and so on to match you up with a plan? My husband just said that MarkLogic appears cool (for some government honchos).

MadisonMan said...

If the Republicans take the Senate & hold the House, expect hearings on this topic.

Dear God, no.

The Republicans should -- when they take control -- simply fix the problem. Pointing fingers at all the horrible things that Democrats have done might give a short-term schadenfraude-ish frisson, but long-term? The optics are that it's just business as usual, meaning the pendulum will swing the other way eventually, and Republicans will be swept out after having done nothing (except vote against Abortion and against Gun Control) substantive.

What's the quote? Republicans say that Government does nothing, then they get elected and prove it. (I can't recall at the moment the Democratic yin to the Republican yang of that quote)

Carol said...

I work for a govt contractor but we do much smaller systems, and the same thing happens. The agency puts out an RFP saying they want a "commercial off the shelf system" then goes on to spec a thousand requirements the consultant gathered from every vendor in the market. Then for good measure they say it has to run on Microsoft SQL.

So an honest bidder has to say no to a bunch of shit and then risk not getting shortlisted. Which is what usually happens to us.

Winning the bid seems to be an even worse outcome.

Ann Althouse said...

"Well obviously code is used to patch flaws. But the problem here was likely whether using hacky code as a shim to repurpose already written code when a higher level redesign was called for. Every software engineer reaches that decision point sometime in a project, and it's a test of character for both the engineer and whomever is managing the project."

Yes, but you're reinforcing my point, that it was paranoid to think that the patching approach was a hiding of the flaws. It was just perhaps not the best choice of how to fix them. It's one way to fix them, and the doofuses in the govt imagined it to be something underhanded.

T J Sawyer said...

I've no idea if NoSQL was an appropriate choice for this application or not. - Too many years out of the I/T loop. But I would sure like to see the decision criteria and know who made the call! I suspect this might be at the heart of the book that eventually gets written about the project.

One thing I do know - when a company chooses to denigrate a standard like SQL by calling their product NoSQL, the best thing to do is run in the opposite direction.

Joe said...

I believe NoSQL is an idiotic choice for this product. The data here is extremely highly structured for which SQL is a perfect choice and, among other things, has a standard core set.

Developers like saying that you use the best tool for the job. The problem is that "best" rarely includes "being able to hire people at a reasonable cost."

Using MarkLogic not only locks the government into a single vendor, it requires that you hire highly specialized developers or train them at great cost. It weeds out a lot of very talented developers who want to be able to leverage their experience for future jobs.

BTW, for full disclosure:

"Anthony Welters, a top campaign bundler for Obama and frequent White House guest, is the executive vice president of UnitedHealth Group, which owns the software company now at the center of the ObamaCare Web-site fiasco."

So even the argument of "best tool for the job" falls flat on its face.

Kirk Parker said...

pm317,

"Then there is this"

No there isn't! If there's one thing we learned from the Palin candidacy, it's that things that happen in states whose population is the size of Alaska simply Do. Not. Matter. Well, Vermont is significantly smaller than AK, so.....

Zach said...

I'm obviously not an expert, but to insist on a technology that is fundamentally different from the one your lead contractor wants to use seems like the sort of mistake that people kick themselves over.

One issue about the SQL/noSQL decision that wasn't mentioned in the article is the database styles used by the older government databases that the website had to connect to. If you've got to generate totally different queries depending on what computer a request is going to, that would really increase the complexity of the project.

Reading the article, the crucial failures made by the white house were the failure to appoint a project manager in the White House, and to choose a contractor for systems integration.

This is two mistakes, because in addition to a lead contractor, the project also needed someone with enough clout in the White House to resist design changes, to insist that technical people be heard, to deliver bad news, and to either make sure that the project was on schedule or to insist that the schedule be pushed back. Those are all fundamentally political roles.

Reading the article, it's very clear that Chao didn't have the ability to say no to anything, or to deliver bad news to anybody who cared to hear it.

pm317 said...

LOL, @Kirk Parker..agree.. but people must have surely seen the hearings from the hacker guys on the hill?

Xmas said...

I agree with Joe, at first glance the requirements for this project appear to require highly structured data. I mean, a US address is going to be in a US address format. A person's income verification is going to require a very specific set of information.

On the other side, the information each insurer is required to present in each state is going to be different. So maybe the information gathered from insurance companies is going to be tricky. However, that volume of data, compared to the consumer side of things, is so much smaller. I mean, you've got 10 or so insurers, each offering 10 to 20 insurance options in each state on the Federal exchange. You'll end up with a price list of about 3000 records? Let's be generous and add another factor of 10, so 30,000 possible insurance options. This is dwarfed by the 15 million expected consumers, who all have to give practically the same information.

I'll tell you, I wouldn't expect an insurance company with 15 million customers to be using a NoSQL system for customer management. Maybe for business and trend analysis, you could see the customer data and transactional data dumped into a big NoSQL database, along with CDC reports, complaint logging from customer service, other document data and maybe social network feeds. But certainly not for day-to-day operations.

pm317 said...

This is two mistakes, because in addition to a lead contractor, the project also needed someone with enough clout in the White House to resist design changes, to insist that technical people be heard, to deliver bad news, and to either make sure that the project was on schedule or to insist that the schedule be pushed back. Those are all fundamentally political roles.


Sounds like good advice but this WH is afraid of its own shadow and has so many skeletons that to think it would let in a competent person into its circle is not realistic.

The Drill SGT said...

One issue about the SQL/noSQL decision that wasn't mentioned in the article is the database styles used by the older government databases that the website had to connect to. If you've got to generate totally different queries depending on what computer a request is going to, that would really increase the complexity of the project.

A couple of points:

1. The data in the legacy systems was structured and ideal for SQL queries
2. It may have been in a variety of SQL and other legacy or flat file DB's. However, there are translation interface kits, that make the connections routine. Does NoSQL have an IDMS interface, or one to DB2, or even more arcane hierarchical DBs?

AJ Lynch said...

Drill Sgt has the right and simple solution- they should have created a basisc date reference table that was updated regularly. Then they complex interfaces would have been unnecessary and the whole process would have been faster and cleaner and crisper. WTF were these dopes thinking?

Hagar said...

That isn't it, but they have their heads in the clouds and no idea of the practical world.

Or, what Michael K. calls their "cargo cult" ideas of how the world works.

Kirk Parker said...

YoungHegelian,

"maybe we should all get together & put in an unsolicited (to say the least!) proposal to come in and fix up that gol-durned, pesky web site "

You got a good supply of fire and swords? But of course, there's absolutely no possibility of "fixing", the premises themselves are all wrong.

Kirk Parker said...
This comment has been removed by the author.
Hagar said...

And none of all these technical IT discussions address that most of those presently uninsured are blissfully unaware that this brouhaha has anything to do with them and are not going to be much impressed if/when they are told.

The people signing up are those already sick, looking for treatment, and told they have to get insurance first, and those presently insured whose policies have been cancelled and looking to replace them - if they can afford it and do not choose to go cash and carry with their doctor/clinic.

I would think it quite possible that if they get Obamacare going, the country could actually wind up with more uninsured than before.

Leit Bart said...

What do you IT folks think about the emerging backup plan, which is to send the subsidy-eligible folks to stealthcare.gov and send the (mostly) more affluent to enroll directly through an insurer's website?

http://hotair.com/archives/2013/11/20/next-obamacare-dilemma-for-obama-break-the-law-to-let-people-buy-subsidized-plans-directly-from-insurers-or-not/

Strelnikov said...

This article confirms what most reality based people have known all along: This program is based almost totally on magical thinking; on the sheer unstoppable force of Obama's will. "It will work because I say it will", seems to have been the basis for everything that followed.

Jeffrey Tarter said...

"Amazon or Google don’t have to interface with a bunch or legacy COBOL batch systems that were never designed for interactive remote access."

Actually, Amazon interacts with literally thousands of distributors and vendors to fulfill orders from "virtual" inventory (and also manages to handle commissions to affiliates and other tricky multi-vendor transactions). Despite some *very* rare glitches, the linkage is incredibly seamless. It can be done.

Ron said...

Imagine getting hired to do a job you believed you knew how to do and showing up at the job site and finding a lot of tools completely unlike the tools you'd used on all the other jobs you'd done, that were the basis of your expertise. This is the stuff of nightmares!

I've been in that professional position more than once...and got blamed for not having expertise that I never claimed I had!

Ron said...

Buy heath insurance through your portal?

"If you like your Althouse...you can keep your Althouse."

YoungHegelian said...

Buy heath insurance through your portal?

And buy that love swing you've had your eye on at the same time!**

**Male and female model not included.

David-2 said...

Ann Althouse said...
I agree it's surprising that CMS would insist on a technology that their lead contractor didn't like."

They didn't just not like it. They were not familiar with it!


Ann Althouse also said...
"CGI officials argued that [MarkLogic] would slow work because it was too unfamiliar."

I'm going to paraphrase that: CGI officials told administration officials didn't know how to use MarkLogic, but they were told they had to use it anyway.


Yes, and consider that later in the article we're told:

But the Medicare agency reserved the role of general contractor, or system integrator, for itself, even though it lacked the necessary in-house software engineering resources to handle such a task.

So .. they didn't have in-house software engineering resources to be a system integrator but they somehow got the idea that MarkLogic was so obviously the correct and appropriate software to use that they insisted on it against the objections of the principal software developer.

How did that happen? I suspect you need to follow the money and/or political influence trails.

Zach said...

1. The data in the legacy systems was structured and ideal for SQL queries
2. It may have been in a variety of SQL and other legacy or flat file DB's. However, there are translation interface kits, that make the connections routine. Does NoSQL have an IDMS interface, or one to DB2, or even more arcane hierarchical DBs?


That's what I was trying to get at. If all of the relevant databases are structured in one way or another, the queries on different databases are going to be more similar than if some are structured and others unstructured.

Bob Boyd said...

PEBRDAC - Problem Exists Between Resolute Desk And Chair

Bob Ellison said...

Oh, MarkSoft. Or MarkBase. I am so excited.

MaxedOutMama said...

Writing as a programmer, you left out the most important point:
That is a major improvement from a month ago, when it was up only 42 percent of the time and 10-hour failures were common. Yet an enormous amount of work remains to be done, all sides agree.

This thing is not done and the payment systems aren't done, and it is nearly December. There is no possible way the people can sign up in the numbers required to cover even 30% of the uninsured plus those who have had policies cancelled.

The entire article reads like the techies are breaking the wall of silence because of the shitstorm that is about to hit.

So nothing has changed - politics are still running the process straight into the wall.

Insurance reform that results in many people losing insurance they had and NOT BEING ABLE TO GET NEW INSURANCE is such a vicious attack on public welfare that it's unthinkable.

And then, even if 3 million people were somehow able to sign up, it doesn't seem like there is time to substitute something to pay the insurance companies the subsidies.

The worst is yet to come, folks.

wildswan said...

A lot of Democrats, probably a majority, still think that the Obamacare problems are website glitches that Republicans are trying to exploit politically. So more has to happen. It's like trying to get a sidewalk in the suburbs - a set number of people have to die before the expense is justified. Similarly the Democrats are so invested with the idea that opposition to Obama or Obamacare is racist that it will take quite some time to get them to see that the site (and the whole of Obamacare) not working is due to Democratic incompetence. There's no place in their minds for the idea of massive incompetence by their own, especially incompetence which damages their own. But the idea will become more familiar with each passing poll.

The Drill SGT said...

Jeffrey Tarter said...
Actually, Amazon interacts with literally thousands of distributors and vendors to fulfill orders from "virtual" inventory (and also manages to handle commissions to affiliates and other tricky multi-vendor transactions).


I suspect, but do not know that doesn't happen.

I bet Amazon interacts with vendors using an industry standard EDI (electronic data interchange) transaction, not unlike those insurance 834's or X12's that pop up in articles.

So Amazon just outputs a record, and the vendors deal with it. That is not close to having to write a database fetch from a legacy COBOL system at the IRS and do it in a less than 1 sec transaction time and do 10,000 a minute.

http://www.ibx.com/pdfs/employers/employer_resources/ansi_companion_guide.pdf

Mike Bruner said...

I always chuckle at clients who say stuff like "we don't want anything too complicated, just make it simple like (insert Web side of company with billion dollar it budget and years of incremental improvements)."

Those projects rarely end well!

Kirk Parker said...

Jeffrey Tarter,

There are no doubt tricky parts of Amazon's operations, but interfacing with "affiliates" is not one of them--that's all done with purely 100% Amazon code.


YoungHegelian,

"**Male and female model not included. "

Yeah, actually I'm not into threesomes anyway.

Tarrou said...

"Republicans say that Government does nothing, then they get elected and prove it."

Yeah, the last time those lazy fucks controlled the government they invaded two countries, passed Medicare Part D, the Patriot Act and NCLB, created Homeland Security and the TSA and put in two new SC justices.

You can say these things were bad, but it's a stretch to say they're "nothing".

MattL said...

The single Federal website seems like the only good decision that was made. I've always been baffled why we needed to have all of these different sites to begin with, since this is a Federal law. They're all doing the same thing. Why waste money building 50 buggy sites when you can waste money building just one?

Bruce Hayden said...

There are no doubt tricky parts of Amazon's operations, but interfacing with "affiliates" is not one of them--that's all done with purely 100% Amazon code.

I think dealing with Amazon is probably similar to dealing with Wal-Mart, that apparently pays net 120 (I.e after sitting on the invoice for four months), and even then in some cases almost a month later. You do it on their terms because that is where the money is. Don't like giving Wal-Mart the interest free use of your money for four months? Maybe you can try JC Penny's, which is falling apart, etc. In some businesses, you deal with Amazon on their terms because that is the only real way to make many of the companies doing so survive financially.

Now, look at these other govt agencies that are interfacing with healthcare.gov. IRS? Maybe 80 years old, and interfaces with other agencies and millions of taxpayers on its own terms. Ditto for SSA, etc. Healthcare.gov is the new kid on the block, and behind the scenes would likely be laughed at if it had the presumption to try to force these much older agencies to interface on its terms. Silly them, not understanding that the mandarins have to be appeased if you want to get anything done in the government. Even if those working on this project had been able to get a White House mandate requiring these agencies to provide Healtcare.gov in the format it wanted, it likely wouldn't have helped - because then these agencies could passive aggressively sabatague the project by just not getting the work done on time, and if questioned about their failures, could, and would, point to the absurdity of the requirements. This is what a lot of the higher up career people do in the govt - cover their asses, and do what they want anyway. And, they have become super-graders, in many cases by being good at this.

Bottom line here is that Healthcare.gov was stuck getting the data however the agencies providing it were willing to provide it, and should be happy to be getting it at all.

Bruce Hayden said...

I think, in the end, the rollout of healthcare.gov failed miserably, and, indeed was destined for grand failure, because the top people in the Administration, as well as the Dems' Congressional leadership were clueless. They were almost entirely liberal arts grads, with maybe a law degree, which here just dug the hole deeper. Their limited experience was in govt, where IT projects struggle on for years, and massive cost over runs are just another part of the graft. They have no experience getting large IT projects done at all, and certainly not on time. From what he has said, Obama apparently didn't think this would be much worse to implement than the web sites his zealots set up the last two Presidential elections. How hard could it be? After all, he had acquired the Presidency with little work or preparation, except the writing of his autobiography. Setting up this web site should be child's play for his brilliant team.

In short, the didn't know, or even have a clue about what they didn't know, and as a result made all the classic IT mistakes, but at a level probably never seen before.

cokaygne said...

I have been away for a long time, but many of these problems were solved by state Medicaid and welafare aganceis a long time ago. Not building a separate web site for each state was a major mistake. Even if the state's governor decided not to expand Medicaid, a decision which i support because Medicaid expansion is like the Hotel California, you can check in, but you can NEVER check out, they would have been able to borrow the expertise of state employees who have been there and done that.

Before you grant someone Medicaid or SNAP or cash assistance, you have to verify what they have claimed on their application. You do that by checking with other agencies. It is not rocket science. States have built the work-arounds for interfacing with the legacy systems of other agencies. People at DHHS below the administration would have known this because they would have signed off on those state systems.

Making CMS the systems integrator was a major mistake. Everyone who works at CMS has, or should have, a job to do. They have their hands full working with providers, contractors, state agencies, and beneficiaries. Where are they supposed to find the time to do a proper SI job? You need an SI for whom building this system is job one. You need to meet with that SI every week and go over a list of deliverables. In turn, the SI needs to work with the subs on a daily basis to monitor their progress or lack thereof. I really feel bad for Chao. Chances are he was the one person at CMS whose day job was not very important. If his usual job was not very important, what does that tell you about his clout at CMS? They probably put him in way over his head. It looks like he tried very hard to get the attention of his superiors, but they probably ignored him because he was not a big shot.

By the way, another problem i suspect is that the WH hotshots
are too young to have read Dilbert on a daily basis. Dilbert could explain it all.

Mark said...

One of the many big problems is just the complexity of the law, and the fact that it continues to change. A computer program implements logic that is derived from somewhere, and that somewhere is the 418,779 words of the ACA, and the somewhere between 10,000 - 20,000 pages of regulations that derived from the bill that had to be passed, to figure out what is in it.

The law (and resulting regulations) are not written in a way that leads to a technical translation into a functioning system. As you translate huge, incoherent, gobbledygook words into code, that is what breaks. Sewage in + sewage out.

What we have seen so far is the easy part of signing someone up. The logic will be making mistakes, and we will see lawsuits coming out of the information provided to the users.

When people lose their current insurance on Dec 31st, and they are uncovered on January 1st since the payment couldn't arrive in time due to the website being down, who is going to pay for these people's medical emergencies. To be covered on Jan 1st, you have to pay a premium payment on Dec 15th. For some people, this will be a Medical Insurance vs. Christmas presents decision.

This will continue to get worse and worse.

John Dudley said...

Acetech Information, a leading software development company of India offers Software Development,Custom Software Development ,Website Design, website development , search engine optimization, ecommerce and website maintenance services for its customers around the globe.