She made her comments at a hearing of the Senate Finance Committee hours after the Obama administration disclosed that the chief information officer at the Centers for Medicare and Medicaid Services would retire. His office supervised the creation of the troubled website.Okay, then. Trenkle down. Feel better yet? But let's look at Sebelius:
The official, Tony Trenkle, will step down on Nov. 15 “to take a position in the private sector,” according to an email circulated among agency employees. He has supervised the spending of $2 billion a year on information technology products and services, including the development of the website.
Ms. Sebelius said officials had a list of “a couple of hundred functional fixes” that had to be made so the website, HealthCare.gov, would work smoothly for most users by Nov. 30, a deadline set by the administration.Oh, the punch list! The list of a couple hundred things they've noted need fixing. They haven't fixed them yet. They've just noticed a couple hundred things, in there in that 5 million lines of code. Get on it, code-writing peons:
“We’re not where we need to be,” Ms. Sebelius said. “It’s a pretty aggressive schedule to get to the entire punch list by the end of November.”
Or do you prefer "Look, Daddy! Paste it!"?
Posts
61 comments:
A 'punch list' is a final list of things needing completion AFTER the project 'substantially works'. Since the website doesn't 'work', this is not a punch list. It's a site redesign going on in 4 weeks.
More minimization talk, in my opinion.
They probably got rid of Mr. Trenkle because he had the unmitigated gall to tell them they were full of shit the whole time, and that he absolutely refused to work his people like slaves to salvage the administration's PR disaster.
Just wait and see what he says. This is how it goes when GIS employees & political appointees bump heads.
Word from a very competent high level programmer... me!
The first step toward the creation of a successful website, particularly a data driven website, is the creation of a ruthlessly honest set of requirements for functionality.
The starting point for this website was a 2000 page law and thousands of added on regulations.
No degree of nerd technical ability and cleverness can make something out of this hash, particularly since the baseline requirement was to deceive.
Speaking of "Daddys":
Did You Plug the Hole Yet, Daddy?
What the Democrats fear is that American might be drawing-up their own "punch list".
What conservatives fear is that Republicans will screw-up its implementation.
I don't get it. Is the gold lame clad blond that everyone walks away from?
As one who's sponsored pretty good sized IT projects, the number of lines of code is only one measure of the complexity of the build. Others so far unaddressed include; the underlying data quality, structure and source; the build-requirement documentation; the executive sponsor engagement & alignment; the server architecture; the feeds; the processing modules; the documentation; the PM discipline and network management among others.
Tricky stuff that.
Rumor has it that Trenkle was fired (or pushed) because he refused to sign the security certificate saying the system was secure. This is a LEGAL REQUIREMENT before any govt system goes live.
Instead his boss signed an interim certificate saying that they would fix any security problems.
I suspect that he will be before Congress in a bit. It will be interesting to hear what he has to say.
Unless he dies in a car crash or something beforehand.
I don't know about punchlists in IT but in residential and industrial construction projects they are like the first commenter said.
Minor things that still need correction but that generally do not stop occupancy.
Not, "we still need to put a roof on the building."
John Henry
first line should say "I don't get it. Is the gold lame clad blond that everyone walks away from Kathleen Sebelius" ?
Someone on television yesterday suggested they will keep Sebelius around until the website works, so that all of the shit attaches to her. Only then will they discard her.
“It’s a pretty aggressive schedule to get to the entire punch list by the end of November.
Step one in admitting they're going to miss 11-30 also. At least this time they know what everyone else knows before it happens.
I'm sure this Spruce Goose is going to flying across the Atlantic at ten thousand feet by year's end. Five million lines of code and a couple of hundred functional fixes, sure no problem. Piece of cake.
I hoping for the 11/30 fix!
Get the Titanic back on schedule!
Probably an executive level, and will "retire" with a very nice pension w benefits, now going to double dip with the private sector- aka the Lois Lerner punishment-lose your job for wrongdoing and buy a nice new vacation home.
So, you fix a hundred bugs in the code from a punch list. Have it ready by the end of November. Anyone see a problem with the concept?
Aggressive or stupid?
If there were just one thing to fix, maybe. But, the whole crash was that there was no QC cycle in the process. They tested the site and went live. Nowhere did they schedule time to fix anything if it failed the QC test. So, now they are going to repeat the process. If you change code, it has to go through a testing process. If if change 100 sections of code, the potential QC problems can be exponential as much of it will be done by independent coders, which can create new sections of code that will not work with other new sections of code.
Why would they do this knowing that Nov 30 isn't enough time? Are they betting again that it will be good enough? How did that work out?
Fixing bugs can create new bugs.
as I understand it, Trenkle resigned because they wanted him to sign off that the website met federal security regs for websites. He refused because there hadn't been any testing.
"Fixing bugs can create new bugs."
A website designer on TV a few days ago estimated one new bug for every four fixes.
@Igv,
If if change 100 sections of code, the potential QC problems can be exponential as much of it will be done by independent coders, which can create new sections of code that will not work with other new sections of code.
Young man, if you insist upon trying to speak truth to power absolutely no good will come of it, let me assure you.
What if, as I suspect, the basic architecture sucks?
In my experience, when the word "aggressive" gets thrown out in terms of an IT schedule, what it means is "No Effing Way." I hope I am wrong, I don't want to see millions of people without insurance.
i just want to know how obama can apologize for promises that he has denied making
This is what you get when you mix the peter principle with leeches and derelicts voting to help themselves to the fruits of someone else's labors.
Yea, all of those folks who warned about these kinds of consequences, those "racist haters", what the hell did they know ? Geez. Another prog shit storm, who'd a thunk ?
And this is still just about getting people "signed up."
After that, look at the amount of information about you and the number of companies and agencies that have to keep that information current between them and work together in order for the medical facilities to get paid and the insurance companies to get reimbursed for the premium shortfalls depending on your financial and social status at any one time, and the government to guard against "waste, fraud, and abuse," etc. and so forth.
Ain't gonna happen, nohow!
" Blogger RecChief said... "
The mfm will be along any minute to provide the requisite excuses and redefinitions of the appropriate terminology. Oh, and to tell us how stupid, ignorant, racist, all of the above, we are for having the audacity to question his royal jackass.
RecChief said... i just want to know how obama can apologize for promises that he has denied making
11/8/13, 11:01 AM
Excellent point! I'm stealing it.
I can only imagine what hackers will do to this site. A hasty redesign (I agree, this is NOT working on a punch list) will fill the site with any number of security holes.
Good bye to your privacy. Hello Identity theft, perpetrated from .ru and .cn domains.
We have to sign people up to know if it works
My experience with coding is regulated to Basic and CNC G code, but one thing I learned is if you change one line more often than not ten other lines need to changed as well. Even then it is sometimes simpler to scrap the whole thing and start over.
I'm trying to figure out, from Senate testimony, exactly what happened with the security certification. It sounds like Tavener just went ahead and signed off on the security, even though it hadn't been tested. She just....signed that it was secure.
Is that right?
A punch list with 200 items for a project of this size is not that many, and of course they may have fixed many of them already.
I've got 30 items on a list for software update we expect to ship before Thanksgiving, and no one but me to work that list.
Yes, that list might balloon to 300 to 400 before they get to the quality level they need. But the list never gets down to 0. That's how these lists work.
Tony Trenko should have worked that list with his $400M team before the healthcare.gov website went live, so unless he was screaming to hold the release, he should resign.
I see no serious problem in redesigning the airplane as it heads down the runway toward takeoff. Now if we could only get Obama on board.
And if Tony Trenko was screaming not to release the healthcare.gov website because it was still too buggy, and they released it anyway, he also has to resign.
A punch list is a list of minor and not mission critical items to clean up at the end of a successful project.
Boy did Orwell have it right.
We have to pass it to know what's in it.
You have to sign up to know if the site works.
You have to buy it to know how much it costs.
You have to get sick before you know which doctors to call.
We have to cut you open before we tell you what's wrong.
The failure of the liberal imagination consists chiefly of an inability to imagine that a liberal program can fail. No liberal can imagine that Head Start doesn't work. Ditto with ACA.
@Left Bank, did you write "might balloon"? I was once part of a technical leadership team brought in to salvage a government program where the bugs were increasing geometrically, since each bug that was swatted turned out to have hidden three or four that the code simply hadn't reached before crashing. In the end it was simpler to start over from scratch.
An educated guess: the basic problem appears, from the outside, to be that the web site goes and hits a large-ish number of databases on other USG computer systems. Those systems were never designed for and are not up to this new load on them, so those requests from healthcare.gov fail. The user sees a 404 or whatever and thinks "the site is broken!" but it's really a multitude of back-end problems.
And back-end problems are *much* harder to fix than a busted website. I'm guessing the physical infrastructure of at least a few of those agency databases is inadequate to the task. You can finesse this to a certain extent by optimizing the code on the front end so it doesn't hit the DB quite so frequently, but there are limits. Remember, the website failed in testing with 1100 users. They need something like three orders of magnitude improvement. I'd be astonished if they can get that by fiddling with the code on the front end.
"A punch list with 200 items for a project of this size is not that many, and of course they may have fixed many of them already."
If it filled with punch list type items, then yes, e.g. a text box appears in the wrong spot in IE or something. I'm thinking all the fixes needed have to get done before they can even create a punch list.
"We have to cut you open before we tell you what's wrong."
Actually, I've been to lesser developed countries where this is SOP. When I say SOP, I mean that quite literally. This is what you do when you have no diagnostic equipment.
Well, from my perspective as a programmer, I would not rule out that the "punch list" involves rewriting about 3.5 million lines of that 5 million lines of code, whatever the hell that means. A line of code could have one character on it.
They've been at it for five weeks already, so don't hold your breath.
This is not a "bug-fixing" project, this is a 2/5ths rework project at a minimum, and it's probably more like 3/5ths.
MayBee @ 11:28 - yes, that is correct. I have been following along via the GAO reporting.
I would NOT put my personal info on that website.
For as long as I can remember, the Left has been saying that with modern technology, central planning makes it possible to run things more effectively and efficiently than the clumsy old free enterprise system. I took a course in college from Kenneth Galbraith, and that was the basic message. Fifty years later, and the Left still has faith that technology will make central planning work the way its supposed to. I'm reminded of the religious sects that kept expecting the Second Coming; when it didn't happen on the preicted date, they rejiggered their calculations and started looking forward to the new date. November 30 for sure!
@Shouting Thomas
Yes it looks like a requirements issue. It seems that they waited until after the 2012 election to write many of the regulations to implement the law (so as to not give those nasty no-goodnik Republicans specific things to complain about), so the requirements had to be thrown together at the last minute. Essentially, they blew 2 years of development time for purely political reasons.
I spent many years negotiating contracts for telecom buyers and it seems to me that we have a failure of acceptance testing, and probably a failure to properly write the acceptance test protocol.
Fail acceptance testing = no payment and no payment until the system passes. (Progress payments made as lower level tests were passed, of course, but a substantial percentage of the total price was not paid until acceptance tests were passed.)
Or, it's of entirely possible that political appointees said it must launch on the scheduled date whether it passed or failed acceptance testing.
In punch lists, we always assigned a priority identifier - cosmetic issues that didn't reduce functionality, minor issues that caused inconvenience but could be worked around without serious difficulty and major issues where important functionality was missing or incomplete. There's always a fine line between a condition that fails acceptance testing and something that passes but results in a major punch list issue, and as a representative of the buyer, our position was that if the issue was serious enough to be ranked a "major" issue on the punch list, it meant acceptance testing was failed. Not unexpectedly the sellers had a different view and it wasn't unknown to wind up with a system that had a handful of major punch list items.
Galbraith was wrong.
Fun and games. You aren't going to get rid of bugs in a project this size, but you can graph their frequency on a curve and release when the curve has flattened out enough. Or, you can release before initial testing is completed, after completing 2/5 (or 3/5) of the work, and pray. Usually that means only a couple of early adapters are hurt. Here, of course, it means that the previously uninsured and most who previously were insured individually, are liable for the mandate "tax", and the latter are now uninsured now.
What is amazing is their utter cluelessness. Never having worked on or managed, or even been close to a decently large IT project (in Obama's case - probably never even close). At a minimum, a year's worth of Dilbert should probably be a requirement for this sort of thing.
It is not enough to read Dilbert, but one must understand that every word of it is true.
On December 1, Obama will read the newspaper, and find that the web site still doesn't work.
Monday morning I made one minor change to one line of code. It took me the rest of the day to re-stabilize the code due to the ripple effect of that change (it was worth it.)
Americans landed on the moon a half century ago using computers that have thousands times less processing power than a smart phone.
America used to have smart people, now America has smart phones and unthinking people. The people who would rather believe their lying Dear Leader than their own eyes and ears.
This is not a "bug-fixing" project, this is a 2/5ths rework project at a minimum, and it's probably more like 3/5ths.
...To wit, There is zero chance that rewriting five million lines of code is the answer. Either the solution is a lot simpler or there is no solution other than to start over.
Has anyone seen the "punch list"?
There's a huge difference between "door sticks" and "roof missing" and even the former can be a huge problem.
If it's 2-300 items things like "get {response} sooner", God help them.
" Fifty years later, and the Left still has faith that technology will make central planning work the way its supposed to. I'm reminded of the religious sects that kept expecting the Second Coming; when it didn't happen on the preicted date, they rejiggered their calculations and started looking forward to the new date. November 30 for sure!"
My own analogy is a cargo cult. The lefties are all staring at the sky waiting for the airplanes to come back with the stuff. After all, the control tower looks almost real.
It sounds like Tavener just went ahead and signed off on the security, even though it hadn't been tested. She just....signed that it was secure.
Is that right?
Of course that's right. People in government routinely do things that would get a private businessman locked up for decades, in the blithe knowledge that the likes of Eric Holder will never hold anyone accountable for anything if it would make his boss look bad.
Jack of Va mentioned acceptance testing. I had not thought of it in an IT context but it is something I do frequently with capital machinery.
I even wrote a book about it "Secrets of Buying Packaging Machinery". A bit different than buying an IT project in the details but not that much in the overall concept.
1) Gather info on everything upfront and get signed agreement from everyone
2) Develop good specifications and get everyone to agree to them.
3) Qualify the potential vendors and then request proposals.
4) Select the best one
5) Supervise them during the build to make sure they are on the right track.
6) Perform a rigorous acceptance test
7) Then do the startup and be prepared for problems that have to be resolved. And no matter how well you do the first 6 steps, I guarantee there will be unforeseen problems.
Seems like Obie & Co just skipped right ahead to step 7.
While my book is about buying packaging machinery, it is written to be applicable to buying any type of capital equipment.
Buy it through Ann's Amazon portal.
John Henry
I don't think Tavener signed that it was secure. I understand that she signed an "interim security certificate" which says that it will, eventually, be secure.
Apparently there was no legal basis for her to do so. She may have committed a crime by doing it.
John Henry
You know what? In a couple more years Obama will be qualified to be president, what with all the stuff he has been learning.
For instance, he learned one of the prime rules of software development, "Bad news only gets worse the longer you sit on it."
Making things is hard.
Making them work is harder
I'm not sure this is relevant to this discussion, but I mentioned Galbraith a few hours ago, so here's something else I remember from his class. He told us that through advertising corporations created the demand for their own products, so basically they could sell whatever they decided to produce. This was 3-4 years after the failure of the Edsel, the highly hyped Ford that nobody would buy. This was one of the most spectacular marketing failures of a generation, but Galbraith apparently learned no lesson from it. Ford was not able to impose a mandate that everyone buy an Edsel.
There are some more things to be studied and thought about here.
The Edsel was not a bad car for its time; quite a highway cruiser.
So why the big marketing failure?
Hagar said...
There are some more things to be studied and thought about here.
The Edsel was not a bad car for its time; quite a highway cruiser.
So why the big marketing failure?
(1)The grill looked like a toilet seat.
(2)Henry Ford Sr. Did everything he could to sabotage his son's- Ford Jr his eldest son- success. The Edsel was Jr.s idea from start to finish. Ford Sr. encouraged dealerships not to support the Edsel.
Rusty, at the time (I was a teen-ager) I thought the Edsel grill looked like part of the female anatomy.
Rusty said:
(2)Henry Ford Sr. Did everything he could to sabotage his son's- Ford Jr his eldest son- success. The Edsel was Jr.s idea from start to finish. Ford Sr. encouraged dealerships not to support the Edsel."
Oh, really?
The Edsel came out in the fall of 1957.
Edsel Ford died in 1943.
Henry Ford Senior died in 1948.
At the time the Edsel came out, Henry Ford II, Edsel's son, had been head of the company for almost 10 years.
Unless Henry I and Edsel were fighting beyond the grave it is hard for me to understand how either of them had any influence on the Edsel car.
John Henry
Unless they were fighting
Post a Comment