How Complex Systems Fail (1998)(how.complexsystems.fail) |
How Complex Systems Fail (1998)(how.complexsystems.fail) |
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.
But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.
All practitioner actions are gambles. I should have that inked somewhere.
It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that the failed O-rings were not the single root cause of the Challenger explosion. Another cause was management normalizing the deviations of previous unsafe datapoints of prior launches which let them greenlight the launch in freezing temperatures. Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. It wasn't The Rogers Commission that found the multiple causes; it was NASA engineers and management themselves explaining the multiple causes as they were interviewed by the Rogers Commission members.
For whatever reason, alternative jargon such as "Root Causes Analysis" (plural) or "Proximate and Distal Causes Analysis" isn't as widely used.
In the case of Challenger, I think it's pretty clear this was the "root cause":
> According to testimony by Kilminster and Boisjoly, Mason finally turned to Bob Lund and said, "Take off your engineering hat and put on your management hat." Joe Kilminster wrote out the new recommendation and went back on line with the teleconference.
> The new recommendation stated that the cold was still a safety concern, but their people had found that the original data was indeed inconclusive and their "engineering assessment" was that launch was recommended, even though the engineers had no part in writing the new recommendation and refused to sign it.
-- https://onlineethics.virginia.edu/cases/engineering-ethics-c...
If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.
- by a plain reading it's clearly singular; seeing beyond that requires at least some effort (e.g. curiosity, education... but also authority, responsibility, time)
- there are lots of people dealing with complex system failures that haven't been exposed to any of the relevant theory... tech folk, but also mgmt, comms (which can be more problematic)
- concluding with a single cause is frequently less work
- concluding with a single cause is frequently convenient (e.g. pinning the blame on a single vendor, component, person)
That's why it's helpful to introduce terminology that points everyone's brains in the right direction from the very start, e.g. contributing factors.
(also common sense isn't real)
Not when properly understood--which, to be fair, it often isn't. The proper purpose of root cause analysis is to answer the question: does this failure mean I need to change something about this system? If the answer turns out to be yes, the thing that you need to change gets called the "root cause" of the failure. Of course that's a misnomer, strictly speaking, but that doesn't mean you answered the actual question wrong.
That said, if you look at a system that failed and determine a root cause analysis is pointless, you may have just stumbled upon your root cause.
It's safer to say 'does this failure mean I need to change the system?'. Otherwise you're still facing the single cause trap.
If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/
Basically a poorly designed system is going to fail somewhere. I like to think about it either as the part of the iceberg that's visible or my other analogy is lighting doesn't strike twice at the same place. A robust/well designed system is just very different than a brittle/poorly designed system. The brittle system will exhibit random failures and trying to chase them is counterproductive.
We do have systems that are fairly complex and reliable. Take an internal combustion engine as one example or an entire car or an airplane (well, one of the well designed ones). Then we build bigger systems on top of that, a city's transportation system, where we do get to scales where things "fail" all the time. Yes, some bus somewhere or some train somewhere doesn't get to a station at the correct time, because a bus breaks down or heavy traffic or whatnot. But still a well designed system is robust to those. The parts we build the systems out of are well understood and so the resulting reliability can also be understood.
With software specifically we tend to not design and build that well. We throw stuff together. Then we're surprised when they fail. And we make excuses by saying "complex systems". I often feel it's our lack of discipline and skill vs. other domains and not really that our systems are that much more complex.
EDIT: Some might argue that mechanical systems like engines are just fundamentally simpler than software. But look at something like a bearing. Just one bearing is an incredibly complex system. Bearing failure is a statistical event, some bearings fail sooner, some fail later. There are centuries of know-how in the bearings used in said engine. Mechanical engineers generally use well understood components/designs and build in margin to meet the requirements of the engine they're designing. In software we often re-invent the "bearing" or we'll pick some new unproven design for a "bearing" and we'll generally build without a good understanding of the performance or failure modes of our "bearings" and without a known pattern as to how to use them and when to use them. And then we're surprised when our "car" fails in weird ways. We also don't test to the same standard that a mechanical engineer would use for a new design. They would have a room full of widgets doing ten million cycles before they accept the design as part of a larger system. We "yolo" it...
I will argue that mechanical systems are fundamentally simpler than software, though. Try writing a datasheet for some library as an exercise. Thoroughly documenting the interface is tedious, but straightforward. You need graphs though, so maybe performance under load? But that's assuming a lot about hardware and runtimes, so you need to characterize it under all of those possible degrees of freedom. That's a much larger space than physical components have. Imagine characterizing/qualifying parts under disparate manufacturing methods that include "some sleep-deprived rando who's never heard of a lathe needs to build this successfully in his garage" (i.e. the norm in OSS).
And even if you successfully write a datasheet, some unrelated component elsewhere in the system can remotely interfere with your software through chained vulnerabilities, memory pressure, contention, etc. Mechanical engineers rarely worry about entirely disconnected parts interfering in undocumented ways, because reality limits how things can interact. RF engineers consider this kind of problem more regularly, and need correspondingly more animal sacrifices. The larger the software the worse this problem becomes, and small software systems are increasingly endangered these days.
1. Command in War (1985) by Martin van Creveldt. He dissects may complex but common military failure causes up to the Vietnam War.
2. Team of Teams (2015) by Stanley McChrystal that is focused on what he and teams learned in Iraq but extrapolating to most organizations.
This very much resembles Admiral Cloudberg's write-up of the National Airport collision:
"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."
https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
This is why we created Chaos Engineering. By constantly forcing failure, it made us always create systems in defense of that failure, and gave us great data on where the tipping point is for different systems within a particular failure mode.
> “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” — John Gall, Systemantics (1975)
> All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature.
(Emphasis mine)
I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.
https://en.wikipedia.org/wiki/Normal_Accidents
https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcs...
I disagree with this point. From what I have seen and experienced, there are so many SPOFs in the real-world complex systems that do not have adequate defense or even any defense at all. The only reason catastrophe didn't occur (often) is only because the natural failure rate is extremely low. Like the probability of air traffic controller making a critical mistake is very low, maybe once every few years, but when it happens, the SPOF will almost certainly cause a disaster, and the system (ie. the way air traffic is controlled today) has been around for decades and is so outdated but the cost of changing the entire process around the world is simply way too high so we just take this as "normal".
Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses
Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour
If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.
[1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf
This seems like a list of reasons while I am looking for more abstract directions on how to prevent them.
---
I am trying not to use AIs to just do that for me to tinkle my neurons.
Or so I read the [site? article?]
> Human expertise in complex systems is constantly changing
I feel this is single most important factor responsible to both making system complex and at the same time improving them depending on who the people are and how they take failures and breakdowns. I find it funny but we are in an era where folks building ML systems do not seem to remember the direction in which to open the screws. They do have clear expertise in something new but clearly lack in some other areas
tldr; Removing a process is risky and thankless for most part, no one wants to take that risk. So after every failure a new process gets added and thus you end up with an ever growing complex system
What does that mean? If you knew that the precursors were why did you not set alerts?
E.g. nominally you should never be mixing traffic types (aircraft and helicopters, civilian and military) in close proximity to a major airport and in a regime where TCAS is unlikely to offer sufficient protection. So in theory, any mixing should immediately trigger an alert and investigation to develop new procedures. But in practice, if you routinely allow such mixing under what you believe are "safe" practices (and get away with such mixing for a long time) then when a real accident happens you will have plenty of "proto-accidents" to look back on, but the warning signs from those near-accidents will have become accepted practice.
PS. Btw, I am certain that this is the main cause of the mental health crisis amongst young people.
Its cause we already learned from our failures and make the world a safer place. In my youth we climbed on trees, one of my friend fall down broke his arm, the doctor couldn't fix it and he can not move the hand for the rest of his live. My friend would be happy do not been allowed to climb, or at least under safer conditions but have his hand moveable still. I burned myself really heavy with fireworks as teenager and have a huge scar from it. I never touched any firework after this but also never bought it for my kids, so they never had the chance to "learn" from making the the same mistake. Keep in mind that while claiming the world is too safe you say this from a perspective of an survivor.
Chaos engineering doesn't really apply to data critical or safety critical systems. You can't just break them in the real world to see how they fail.
You have it exactly right -- it has to be modeled and tested in lab conditions. Safety critical systems are not a place for YOLO development.
> (also common sense isn't real)
I prefer Murphy's styled variant:
- common sense isn't
"Something" does not have to mean "one single thing". But I would be fine with your phraseology if it ended up making it clearer to whoever was doing the analysis.
* the number five isn’t magical here. You don’t have to stop at five, and you often shouldn’t.
There's a "system" aspect (outside of power structures and incentives) which is that prob/stats knowledge among almost all engineering disciplines (industrial engineering waves from the sidelines) is exceedingly poor and often viewed as "soft" and "less important" than calculus, linear algebra, etc. And so practitioners are ill-equipped at spotting things like that ("wait, is this the right denominator? what about frequency of incident?").
I think about the things I use daily from my math-focused high school and college career… it’s the social skills I learned in team competitions.
The right level of complexity is the minimum needed to get the job done, and if your existing anomaly detector isn't cutting it...
That's because manufacturing is a simpler environment where causes and consequences are more conceptually distinct.
Until you start to pull management decisions into your analysis, of course. Then it becomes just as complex as software.
Meaning, having something like "management will decide not to fund maintenance for this component in the future, causing a failure" in your fault tree?
You know what they have in common? A lack of complexity. Lovely simple technology, fix it with a hammer. Drive it to the moon and back. Drive it across the desert. Bring tools though!
And then, naturally, you can get complexity just by building stuff on top of that big database server, and other things that depend on those things.
The 80:20 rule still lives.
Could you repair the little Briggs and Stratton in your lawnmower? Almost certainly. You can easily hold a mental model of how it all works in your mind, and diagnose and repair any fault.
Could you actually design such an engine? Where would you start?