- February 11th: Vendor informed of the issue
- February 25th: 28 people die because of the issue
- February 26th: The vendor ships a fix
Feb 21--notice goes out to users to avoid "very long run times". Users do not know what that means, and ignore warning.
https://www.gao.gov/assets/220/215614.pdf (page 9)
"On February 21, 1991, the Patriot Project Office sent a message to Patriot users stating that very long run times could cause a shift in the range gate, resulting in the target being offset. The message also said a software change was being sent that would improve the system’s targeting. However, the message did not specify what constitutes very long run times. According to Army officials, they presumed that the users would not continuously run the batteries for such extended periods of time that the Patriot would fail to track targets. Therefore, they did not think that more detailed guidance was required."
> According to Army officials, the delay in distributing the software from the United States to all Patriot locations was due to the time it took to arrange for air and ground transportation in a wartime environment.
I'm not knowledgeable at all on how software for missile batteries was distributed in 1991 from the US to the Persian Gulf but 11 days doesn't seem unreasonable to me.
It makes me kind of sick imagining that call.
Aren't defense contractors required to be on their toes all the time?
EDIT → Found it. The PATRIOT Project Office.
A more truthful "computer bugs that killed people" example would be the Therac-25 - a machine intended to treat cancer with tightly-focused radiation therapy. Six patients died as a result of massive overdoses of radiation, on the order of 20,000 rads. It was possible for the machine to end up in a state where it delivered full-power radiation without a hardware shield in place to protect the rest of the patient's body. No hardware interlocks were used to ensure that the full power mode was only usable with the shield in place - all safety features relied on software. In addition, the bug was only possible when an operator made a mistake in mode selection and then rapidly (proficiently) corrected it - the rapidity required prevented the bug from being discovered during slow, methodic, careful testing.
See Hackaday's article Killed by a Machine (and associated HN discussion) or for the especially curious, a 49-page post-mortem for more detail:
https://hackaday.com/2015/10/26/killed-by-a-machine-the-ther...
At the time, this incident really stuck out because it broke the illusion of our fabled Patriot missile shield protecting us. Civilian expats really believed the inflated Patriot interception rates parroted to us by mainstream media and our American military expat buddies.
A large number of remaining expats who had stuck out the Gulf War to that point decided to pack it in and leave when word got out that the Dhahran barracks were hit. Although history shows that Iraq surrendered days after this incident, at the time there was heightened fear and confusion amongst the remaining expats, especially the non-Americans.
We left on the last Lufthansa flight (crewed by military personnel) after hearing about this.
Nostalgic edit:
During the Gulf War embassies issued equipment and rations to expat citizens who chose to stay behind. Americans were issued full body suits (for adults and youths) due to the biological and chemical weapon payloads that Saddam boasted his SCUDs were carrying, along with MREs that tasted fabulous! In stark contrast, Commonwealth citizens were issued a bare gas mask (adult size only) and mono-flavour MREs that tasted like cardboard.
The British embassy sticks out in my mind: with stern stone-faced expressions they admonished us all for not evacuating and thus endangering children in a war zone. In addition to the terrible rations and gas masks, they wordlessly gave us a stack of translucent stickers. When asked what they were for, embassy staff explained that in the event of the air siren going off, we should get under our sturdiest tables and don our gas masks (standard procedure), and then slap the stickers on. If the stickers changed colour, it meant we were in the presence of a biochemical agent and would have approximately 10 seconds before we died a horrific death.
You kind of had to be there to appreciate the grim humour.
1. I remember hearing the system was only designed for XX operational hours but was being run over the operational spec.
2. The time was stored in base 10 so the calculation errors added up over time or something like that so if they had used some base 2 timing scheme it would haven't have had issues with rounding errors.
My class was in the mid nineties so the details of my 25 year old memory is pretty hazy...at best.
http://catless.ncl.ac.uk/Risks/13/35#subj1.1
http://catless.ncl.ac.uk/Risks/13/76#subj8.1
And in 1997:
If an EMT fails to save a victim of a car crash, did he/she kill the victim? If the dispatcher misspoke and gave the wrong cross street, delaying aid, did the dispatcher kill them?
You can use a gravitational map that only accounts for latitude, but it isn't as precise.
So using an accurate clock is really important if your intent is to hit a missile with a missile.
[0] http://www.slate.com/articles/news_and_politics/war_stories/...
It now makes much more sense to me that a (terrible) mishap had occurred and possible prevention was only a reboot away. I can see how being exposed to that context at upper levels could easily cause one to latch onto any perceived preventative measures.
I also once saw a short ntp time step across multiple clusters (yeh, simultaneously) shut down half of a wafer factory.
Time is important.. but rebooting all your systems at midnight probably will not help you to control it. This especially if there are large, hot, fast objects flying around in the night sky and definitely, really, don't do ALL of them at the same time every day .. especially during, you know, battle. /pro-tip
Mods should change this. The "software fix" was a software patch which corrected the clocking bug.
The "software workaround" to use pre-fix was reboot.
I hate editorialized, lying titles :(
I don't mean to second-guess them in an area I know so little about, but if that was enough to cause a serious issue in the span of only a few days, shouldn't the devices be designed with a separate synchronization system, at least as a backup? Maybe GPS?
Which brings up a sort of interesting question...would a Patriot missile system even have receivers for a weak public signal like GPS, or is it all self-contained?
But the rest of your point boils down to ’if you know your system has a flaw why not mitigate it’? But of course at design time they didn’t know it had this flaw.
i'd assume it could, since GPS is military, and a mobile missile system is exactly the sort of thing that wants to know where it is, so would have the keys to the (higher resolution) encrypted GPS signals as well.
There is no 1 answer, this argument is a result of black-white/yes-no/us-them single point of blame thinking. and it's terrible.
the bug contributed to the loss of life.
This is a strictly technical examination of the proximate cause of their deaths; it makes no claims about their ultimate cause. Whether or not a missile system with an accurate clock might have hit the target, it is unambiguous that this one missed specifically because of clock drift.
I feel "preventable deaths" is a preferable focus over "cause of death".
While most of us were cowering under our desks and tables during SCUD attacks, some of our American civilian friends were out with their families in the desert trying to film the Patriots "intercepting" the SCUDs and driving out to try and pick up pieces of debris.
I look back upon those days with fondness and gratitude, especially for the American forces that served.
This just isn't so. Also the degree of acceptable reliability that is reasonable is different in a missile defense system vs the toy your grandma uses to browse facebook.
It had to be rebooted because a bug caused it to be increasingly inaccurate the longer it was booted up. This was always broken. It wasn't an acceptable fix because you manifestly can't trust users to do so as shown by the 28 corpses. It was however probably the best that could be done on short notice.
Taking 60-90s completely out of protection to reboot a critical defensive system when someone might, at any moment, toss a Mach 5 projectile at you from a couple hundred miles away is a far-from-ideal fix, even if it had been communicated properly to the end users (which it wasn't.)
https://embeddedgurus.com/barr-code/2014/03/lethal-software-...
Well, wait. It wasn't the explosion - technically, it was the impact of the pressure wave on their bodies that caused ... well, no. Really, it was the fact that their organs stopped working after impact of the ... well. If you really want to be accurate, it was the fact that metabolism ceased to be practicable after their organs stopped working.
Well, no, actually, the fact that their mental processes depended on their metabolism - that was really the cause of their... Well, no...
I know what that's like.
About 20 years ago I was at a consulting firm supporting an electric and gas utility company. Among other things they had to do something called "markouts" which means they paint the ground at a location in a way that indicates exactly what infrastructure they have in the ground and precisely where it is. Markouts are a government organized thing. Before digging somewhere you can call a number and anybody that might possibly have infrastructure in the ground anywhere near your dig site is required to paint their markouts within a short time period. There are stiff fines if you "miss a markout."
Anyhow there was a data problem with a markout. The field worker was sent to paint a markout at the corner of two streets that actually ran parallel to each other and didn't meet. Instead of calling it in and questioning the task he did nothing. Shortly after a construction worker put a backhoe through an electrical conduit with 15K volts. There was an explosion that was heard for many miles. The worker died the next day. He died painfully.
> so I just cannot help but be curious as to how one of those meetings would go.
Finger pointing, of course. Data was being fed back and forth between systems and eventually somebody else took the blame. The field worker who ignored the markout also was blamed. We did add something to our system so that that kind of data error would raise an exception.
I learned a lot about care and diligence about data from this experience. Data errors are no joke.
> I'd have loved to be a fly ...
loving anything about that sad scenario seems impossible.
That's like saying "It's fine, the critical vulnerability patch will be applied on reboot", while in reality all your users just suspend to disk and move that annoying reboot nag window behind the task bar where it's out of sight.
Patriot, in the role deployed in that Gulf War, is a “not being killed” device, but not really a killing device.
0. https://www.sto.nato.int/publications/STO%20Meeting%20Procee...
1. http://users.cecs.anu.edu.au/~Jonghyuk.Kim/teaching/Inertial...
"However, the timestamps of the two radar pulses being compared were converted to floating point differently: one correctly, the other introducing an error proportionate to the operation time so far"
The code had a defect that effects its aim from turning it on but because it took 100 hours to drift by 1/3 of a second the problem wasn't apparent when rebooted regularly. If software can't continue to do basic math without manual intervention its defective.
In fact everyone including the company that made it admits it's defective.
Its possible your teacher picked a great example to illustrate a communication failure.
http://archive.gao.gov/t2pbat6/145960.pdf
Page 2
dug into reference 48 from Wikipedia which referenced this article which I did a search on google.
It is however a good explanation why it remained undetected.
Arguing about whether or not something is a bug is pointless precisely because someone will just pull the "behaviour outside of expected use is undefined" thing out of the bag. Regardless of whether or not you should have expected something to work, if your product unintentionally kills people due to a software error, you have a gigantic problem. It's really that lesson we have to keep in mind.
I get this all the time from project managers: it doesn't matter if X fails because we aren't designing the software for X. But you can't just dismiss X -- you need to understand the consequences of X just in case somebody tries to do it. For example: It corrupts the DB if 2 people edit the same record at the same time. The project manager says, "Not a problem. I got sign off from the groups using the app and they promise never to have 2 people working on the same thing. Problem solved, and no need to modify the code!" Of course a week later the DB is corrupted and it's not a bug (it's a feature ;-) ).
It does make software development more costly, and you need to draw the line somewhere. This requires balancing risk. But I will argue that if you are writing software for a missile, there is no hiding behind the "we didn't design it for that" argument.
Its entirely possible to have any reasonable degree of precision reasonably required to the limits of our tools to measure.
This isn't about an inherent limit of computation its just programmer error.
Generally KPPs (which aren't requirements themselves, but influence the requirements for systems) are set at lower bounds, not upper bounds, for somethnig like this. You wouldn't set a KPP: Should only work for 4 hours. You'd use: Should work for at least 3 hours, 4 hours desirable (or some similar language). If it works for longer, that's great. But longer won't be tested since it's not a requirement or goal for the system, which also means failure modes for longer runtimes won't be encountered because they're outside the bounds of the system requirements and specs.
So, while it's horrible in results, it can be very easy to understand why basic functions would have specs not at all adapted to the use to which it was being put.
Similarly, with regards to the speed of the missiles, the requirement would not be: System SHALL fail to detect missiles above some threshold speed. But rather: System SHALL detect missiles below some threshold speed. This leaves open the possibility that it may be more or less accurate outside that range. It should be documented for the operators as a potential for failure: System may be ineffective against missiles operating above X m/s. But the requirements wouldn't include that detail.
This pushes the problem into the documentation and training. Since it was originally designed as a mobile platform with short run-times, there was no explicit operating procedure requiring reboots. It was just assumed. At the same time, the failure itself (after 20 hours) was unknown because testing hadn't been done to see what would happen.
You set your performance parameters (have some success rate while operating continuously for up to 4 hours). Then you launch missiles at it (simulated and real). If you stop enough of then you’re good.
Wikipedia didn't exist when I was taking the course. It's probably in one of the 100 odd source articles since it wasn't just my professor that pointed it out. One of the other commenters mentioned a similar discussion from one of their professors.
1. I remember hearing the system was only designed for
XX operational hours but was being run over the
operational spec.
I googled up a copy of FM 44-85 to refresh my memory and write this post, its pretty much as I remember it.
The doctrine in chapter 3, planning, is extreme mobility and rapid hour to hour activation and deactivation of individual missile batteries, kinda like infantry bounding overwatch but glacially slower on an hourly basis, for example see Table 3-2 where the four batteries are rotating on and off and moving/maintaining on detailed hour by hour basis, so the doctrine seems to be uptimes should typically be on the order of 3 or 4 hours maybe. Not a zillion days in a row as actually deployed when the software bug hit.
The doctrine in chapter 5, operations, goes into a big discussion of defense design strategies. The weapon system is inherently sectorized this naturally leads to overlapping areas of fire being very important. You have to ask why the unit that had a ridiculous uptime never shut down to perform daily maintenance which would inherently involve rebooting stuff, its no big deal to down a system because sectorization and overlap is inherently built into the technology. Its reasonably well understood that technically you can tell an individual infantry soldier to guard a post for 100 hours or 1000 hours continuously, but someone screwed up if they issued an order like that because its simply impractical. That leadership failure will be discussed later. So... aside from the question of why the software failed under ridiculous conditions, you have to ask WHO more or less knowingly misapplied the resource without backup or planned maintenance intervals? Possibly this section of the FM was rewritten between the tragedy and the the release of the copy I have access to, but its still poorly written. Or what section of the FM would have ever given the officers the idea that the weapons system can be deployed the way they did it? The idea that the weapon system could do what they told it to do came from somewhere and it apparently was not the documentation?
The doctrine in chapter 6, support, has a little blurb about battalion level staff officers. What did the EMMO think about keeping a patriot booted up and running for 100 hours without a maintenance interval? Missile maintenance is literally his only job. And if that slot was unfilled, its the job of S4 and the XO to cover or reassign someone or otherwise work around. Around page 6-16 there's a discussion about operators being responsible for maintenance... I had a humvee assigned to me, I hated it, it leaked oil all the time, but the point is even my junky humvee had daily maintenance tasks for PMCS. The patriot missile PMCS checklist is probably classified, but if a lowly humvee has daily maint, how can a missile not have a much longer and more complicated daily maint? And this implies someone is pencil whipping maint (I mean, everyone kinda does that, but..)
Its hard to summarize a class discussion but from the point of view of a systems analysis class, mostly non-military other than myself, the end users were being innovative and adapting and overcoming which unfortunately means the doctrine and specifications of the weapon system have little to do with how its being used. The class considered this the biggest systems analysis mistake of the tragedy. Why even write docs and specs if the users won't read them and they have no relation to what the users want to do? I guess a good HN analogy would be you could creatively deploy binary executables using the "cat" command and hand typing unicode and that would be a nifty hack to work around a problem but would be a pretty stupid way to operate normally. Specifically the Army's own docs used to train and plan operations imply shorter operations terms interspersed with maintenance intervals and deep redundancy, none of which seems to have anything to do with the failed deployment.
There was a big argument in class that it had nothing to do with systems analysis and was merely a leadership failure, using the example above of technically you can order a soldier to stand guard for a hundred hours, when guard shifts are normally a couple hours, and when he passes out asleep around 48 hours into his shift, you can try to blame the soldier or declare there's a bug in our brain preventing 100 hour deployments, or you can even blame the manual and the technical writers for not putting a warning in the manual not to do dumb things, but fundamentally thats just passing the buck that it was a failure of leadership to assign a unit to a task its not designed to handle, then cover it up by pretending its merely a software bug or something. I don't know enough history of the tragedy; its possible the Army correctly relieved some officers of command and its only the media and press who blame the software bug.
You can imagine the look on the face of the software developers when they got the bug report; like dude, did you ever read FM 44-85, or if you aren't reading it, what are you reading, so we can read it?
https://dl.acm.org/citation.cfm?id=807721
No real good excuse for not actually testing systems that can take or save lives.
http://archive.gao.gov/t2pbat6/145960.pdf
Page 2
I never did embedded programming or government programming, so what your saying make sense from a spec perspective.