Famous outages along with deep postmortems? I just finished reading the Roblox outage postmortem (trending on HN: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/). I learned a lot reading about it and found myself thinking "surely there's other well-written postmortems that are a doozy". Does anyone know a good resource where I can find a compilation of such outages? If they're classified/tagged then even better :) |
0: https://www.ieso.ca/en/Corporate-IESO/Media/Also-of-Interest...
OUTLANDISHLY-GOOD reporting: https://www.ntsb.gov/investigations/AccidentReports/Pages/Re...
They crashed the microsoft.com cluster only days before, sending certain accepted metrics re: uptime from 99.999% to 97.312%.
The cluster crash was caused by some errant out-of-band JavaScript being published to a live MSCOM cluster. The postmortem was not in-depth, it was a cremation. Burn and hide the body.
I thought it odd that all those involved ended up at AWS shortly thereafter, including the executive whose head rolled right out of 1 Microsoft Way.
Those involved owe Dave Cutler an apology, with or without the conspiracy intact.
The second one was a 30+ hour outage that forced us to really up our incident management game.