Two years ago our team wanted to buy a small cluster (~300 cores, ~$50K). We talked directly to two good vendors (good recommendations from university partners) and came up with a fine machine and 2 bids for it. Sent recommendations to procurement.

Procurement put it out for bid, and a fly-by-night company undercut the bid by $10K... by noticing that procurement had not specified details of service level (that were in the bids we'd gotten and forwarded). Procurement, once it goes there, is a true black box. No communication, no understanding.

Five months later, we were basically delivered 2 pallets of unassembled parts and no instructions. Believe me, we spent 3-4x as much in labor as the $10K savings to get it working, and it's been plagued with issues that would have been under the onsite service warranties for the better companies.

The biggest irony is: I firmly believe that procurement acts this way not because the government is fundamentally incompetent, but because the Public, and thus Congress, BELIEVES we are incompetent, so puts so many levels of "check" bureaucracy in place that the people who know what they want can't participate directly in getting it.

Here is the actual, original source for the Waterfall approach, first published in 1970:

http://leadinganswers.typepad.com/leading_answers/files/orig...

If people would just bother to scroll past the first couple of pages, they will notice that the approach already includes some iteration cycles between steps.

In other words, this whole "agile vs. waterfall" debate which has wasted countless hours of human effort is based on a complete misunderstanding of what "waterfall" is in the first place. No one ever seriously proposed a model without iteration. It simply never existed in the first place!

The answer was any time you are developing software for the government! The professor specifically mentioned it in lecture once, so that alone was enough for full credit on the question (other reasonable answers were fine too).

Later I TA'ed the class twice and made sure to eliminate these pure lecture-attendance-check questions.

  while (true) {
    // Contractor working for a year
    Government: This isn't what we wanted!
    Contractor: We met all of the requirements... See all of the boxes are checked.
    Government: Well we want to change 1,2...n things.
    Contractor: Okay, Let's do a follow on contract.
    Government: Okay, Here is the money; Go.
  }

Any market based solution website has to be very agile and responsive [edit: to succeed at it's goal] but the government can't be super responsive and in many ways shouldn't be super responsive. The state spends all of our money and enforces mandatory decisions concerning our lives. The state shouldn't have the agile qualities needed to produce the beautifully flexible websites created by the private sector.

In general, I'd claim the state should certainly be smaller but that it shouldn't be less bureaucratic, shouldn't be more like a corporation. Civil service is boring and bureaucratic by design, specifically it was created to combat the "spoils system" that plagued the early American state [1] (though the prize of modern state eclipse what Tammany Hall etc could have imagined). Modern corporations are agile by having a command structure which lets them quickly maximize profits - which is great if we believe the market system benefits everyone when operating properly. But states with the ability to trample the fences of ordinary market shouldn't not be also given the ability to move quickly and agilely to do this. Corporations have no internal limits to their "greed" but we citizens of democratic market capitalism are assuming that's OK, indeed desirable, as long as the corporation face the strong external limit of markets and individual choice.

The current fashion for what could be called "state-enforced private consumption" is sold as giving us the best of all possible worlds but in reality gives us the worst (IE. the reality is the wealth of this society is indeed being vacuum-out by a kind of private-public rent seeker limited by neither the traditional bounds of the democratic state or traditional bounds of the market).

Note: I'm not a conservative rooting against Obamacare. It seems like it was a terrible approach for achieving affordable healthcare but I still would prefer it succeeded that failed because, well, I and many friends need it.

[1] http://en.wikipedia.org/wiki/Spoils_system

http://digital.cabinetoffice.gov.uk/

Aside from creating https://www.gov.uk/ which laid down a lot of principles on how to fulfil a government contract (as well as the foundations of what goverment websites should look like and how they should be developed), GDS is also looking at the problem of procurement.

The GDS team essentially are wrestling back from the big contractors the major contracts, breaking the work down into a large number of bitesize contracts and then farming them out to a wide variety of smaller vendors.

So instead of finding a Fujitsu/Siemens JV team, or an IBM Professional Services team, operating a £50m project, the plan is to offer 100 x £250k projects to a large number of smaller suppliers instead. Each project having a clearer purpose that is more able to be fulfilled.

Of course there are obvious overheads in managing so many projects, and of course some of these projects will fail. But... overall the savings will be such that the overheads are cheap, and the failed projects will only have a smaller impact on a major programme initiative than a failure would today.

I work on projects that are probably on par in terms of complexity. We typically only involve a handful of firms. And even then, coordinating them all is a challenge. I can't fathom making the process work with 50+ firms.

Maybe that number was hyperbole. I don't know. But if it's true, I shudder at the thought.

I work as a contractor for the Australian government. I personally know of multiple project failures in the 10s of millions of AUD range and a few in the hundreds of millions. These stories don't even make the news.

I've worked at small companies, research institutes, universities and now in government. I've not worked big corporate but have heard that it is similar, although more efficient than government. Size means you get less feedback on what is really useful.

Government fundamentally lacks feedback on what really matters. In the US the department of health cannot be driven out of business by another department that does what is important 10% better. In private industry that discipline and feedback makes things work.

If you build a widget X and it isn't something that people want you go under. That doesn't happen in government. If you build a donkey but it's the donkey they paid for it could be in service for 20 years.

It's hard to see how to make it all better. Perhaps trying to keep components small and having multiple groups build them and select the best might help. Then at least 2+ groups would have to compete to build a better system.

The way they work is purely in water-fall project management mode. Project managers are gods and spend insane amount of time on ms-office calculating hours per each task that are 2 years out. then they bring CGI indian sub-par programmers on L1 to save on costs. Technology is least of their concerns since its about shipping code. Also blame shouldn't be just on CGI, the government is at fault as well. Simple request for information would take about 4 biz days to get it. everything is slow and the Gov IT staff has no clue on how to scale. Anyway, when I heard CGI won this project, I knew it would fail.

And even the software for info systems can have dangerous consequences. Does anyone remember the underwear bomber, who almost brought down a plane and caused a nice surge of invasive security measures afterwards? His own father exposed him, but the State Dept's visa system failed to find the terrorist because someone misspelled his name when entering it into the system

http://www.cnn.com/2010/POLITICS/01/08/terror.suspect.visa/i...

Think about it...the State Dept has been dealing with foreign terrorists well before 9/11, whose names are easily misspelled by Westerners...there's not even a consistent way to spell Osama bin Laden, depending on you interpret he phonetics. And yet no one thought that a fuzzy spellcheck would be useful, apparently. And a whole bunch of people almost died because of it (and the security apparatus greatly increased)

You have to treat prototyping as part of the requirements gather process. Then, when requirements phase is done, you have to treat "development" as really "testing". Because, for the types of clients that are going to insist on a Waterfall project, the final testing is really only a cursory user acceptance testing and they really don't have the skills necessary to determine if you've met their requirements or not.

https://www.gov.uk/transformation (fully responsive design pages together with new service backends delivered with Agile / Scrum)

and the teams doing it: http://digital.cabinetoffice.gov.uk/ (they are hiring more than a dozen people in the moment)

I will point out, however, that there's a huge assumption lurking in there that wasn't explicitly stated: somebody on the government side has to know what they want and be willing to take the heat if they get it wrong. _This_ is the reason so many agencies prefer waterfall -- there's enough obfuscation and paperwork involved that when somebody complains, and in high-risk projects there'll always be complainers, nobody is really at fault. The coder guys can point back to the designer guys. The designer guys can point back to the requirements guys. You'd think that the requirements guys, the guys at the front of the waterfall, would catch all the blame, and they do. But they just write bug tickets because some aspect of the process wasn't followed well enough.

You can spend hours or days trying to figure out what went wrong and not know anything more than you did before you started. Which is exactly why the system has evolved the way it has.

I hear a lot more government projects are going to be Agile. Here's wishing them luck. If done correctly, Agile will 'debug' the organizational problems that lead to this bad performance over and over again. If they just sprinkle a little Agile nomenclature on top of things, it won't do anything at all.

Get that down, and then get several neighboring communities--again, HOAs, neighborhoods, or towns--and get them to adopt your ideas as well. With that amount of variation, you've got a strong base from which you can convince a major city, or a county, to adopt your ideas: after all, many of their constituents are already on it and can endorse it.

This isn't meaningfully different from founding a startup taking on governments as clients.

I've led or worked on tech contracts and grants for ED, HHS, NSF, CDC, and others. Several people have pointed out some important points that are not getting enough attention in my opinion:

by @mcone: "The procurement rules were designed for that, yes. But in real world scenarios, those rules effectively do exactly the opposite. Since there are so many hoops for potential vendors to jump through, only the most established players get to bid on most contracts. And in my experience corruption and cronyism is still alive and well in federal IT contracting."

It's true, incest between government and industry is rampant and has led to wide spread cronyism despite the system's best efforts to limit the effects. People that once worked for company X, now serve on the proposal review panels when company X competes for work. No, they don't receive direct compensation and thus there is no immediate conflict of interest, but the reality is humans are drawn to (or don't want to disappoint) the people they know (former colleagues) and thus pick their old companies. In addition, they know there is a chance they may once again return to said company (so there is long term conflict of interest potential).

Another point that hasn't been discussed is how the government's procurement process provides next to no incentive for companies to efficiently produce good products. If our industry loves the DRY concept, everything about the gov procurement process points to a !DRY (or do repeat yourself). We built and rebuilt the same database for offices of the government that shared a building with one another. But because everyone is in a silo, they don't collaborate well and don't realize that they could pool their needs to develop more universal products. (And on the industry side, as long as gov continues to work this way, they don't even have an incentive to re-use their work or propose innovative, generalizable solutions.)

And for those of you that might say, 'but can't you win a gov contract by bidding lower by working off of existing work?' The truth is price has very little to do with who gets selected for a gov contract.

if government were competent stuff like capitalist economy and privatising public services would always be bad (provably - actual proof, not just overwhelming evidence - but there is overwhelming evidence too).

leaders intuit things like agile and don't label them - instead of picking it up from a book and implementing it badly because they don't understand first principles.

Isn't this the essence of the critique, though? Its not the lack of iteration, but the logic of the spec-formation.

To put this another way, you are right that it has nothing to do with the waterfall model per se. But it has everything to do with accepting the same set of behavioural assumptions. Namely, that the people who spec the model have perfect foresight regarding the spec itself.

What needs to be accepted is "incomplete spec". The problem with this is then the hold-up problem. you get held to ransom to fill in the incompletions. So what is really needed is a capability to execute this more in-house. This would prevent the re-negotiation of the economics (the hold up), because the project manager would just execute properly, directing resources (already paid for) through fiat rather by then re-negotiating vs a modified spec.

One problem with this is accountability. There would need to be more accountability, because execution of the incomplete spec will not be the same as farming responsibility for the spec out to a third party vendor.

Project managers need to get used to the idea of working intensely with a development team rather than asking for a specific thing and then walking away

Quoting Dr. Royce:

> Management of software is simply impossible without a very high degree of documentation.

How much documentation?

> In order to procure a 5 million dollar hardware device, I would expect that a 30 page specification would provide adequate detail to control procurement. In order to procure 5 million dollars of software I would estimate a 1500 page specification is about right in order to achieve comparable control.

5 million 1970 dollars is equivalent to about 30 million 2013 dollars, so for a $30 million project, his advice is:

1. Write a huge stack of documentation. Literally stop everything until that documentation is written.

2. Get feedback from reality and from your stakeholders exactly once.

Sure, having one opportunity to act on feedback is better than never having any, but it doesn't fundamentally change the process or the risks involved.

What the Agile folks realized is that reliable software development is an experimental process. If you only do one experiment, you're still missing that point.

This is provably false by visiting any number of old, large company websites, esp in healthcare or banking. In addition the government isn't in this kind of business, it outsources almost everything to "free market" companies, who extract as much margin as they and their lobbyists can get away with.

Complexity has much more to do with size of and number of evolved entities than whether they are profit motivated or not.

The public reason for this crap is that the system is supposed to be structured to make sure that accountability exists, favoritism is minimized, and employment is boosted (by spreading out the work). The reality is that the system is designed to maximize the enrichment of the lobbyist-connected owners of the contractors.

It's all about organization, and trying to have just enough people to get the work done and nobody more, and the right people, and this has nothing to do with open or closed source.

The government, that is, HSS's CMS, took on the role of integrator, including integration testing (perhaps "Prime Contractor" as mentioned elsewhere). They aren't known for expertise in this (the Pentagon can do this with medium sized weapons projects, which are a rather different field anyway), and ... really screwed up:

They and those above were late with specifications and requirements, kept changing them (7 major ones in the last 10 months per the NYT), were making changes in the week before launch, and when they did a simulation test of 200 simultaneous logins just before launch the modules locked up. As did the site shortly after its midnight launch.

Oh, yeah, three days after the launch CMS panicked and proposed to fire Quality Software Services Inc. (QSSI, a unit of United Health Group) and punt their identity backend system (based on an Oracle package that's known to work), but eventually decided that would take longer than QSSI getting it to work. Who knows, but that's another sign of CMS as the integrator failing hard while distracting both QSSI and CGI Federal.

Now, maybe it ended up being too many cooks because CMS didn't provide strong oversight and coordination, but....

Thinking a little more about the situation, consider that the majority of startups fail. I would not be surprised, if one were to take a look at the success rates of internal projects in large companies, to learn that those are fairly low as well. So if the private sector is more efficient than the public sector in IT, the differences are probably more subtle than one would at first think. In both cases you have lots of capital being spent on projects that won't come to fruition. Perhaps the incentives in the private sector are a little better aligned towards a successful outcome.

I'm sure that system would blow up in some other way, however.

Also, I didn't explicitly say so but small companies are more efficient. If they are not, they tend to go insolvent. That's what is so good about them.

The campaign's site is fluff.

Governments are huge entities, with laws that look like they're there to protect public trust, but are really written to often reward key players (for this area of government).

Pretty much the opposite of a government project.