Heroku: a follow up on last week's outage(status.heroku.com) |
Heroku: a follow up on last week's outage(status.heroku.com) |
An admin was doing a rolling restart that triggered a bug in the loadbalancer software. The auto restart script turned out to just make things worse by restarting it over and over (they always do), so we thought we'd just quick throw spare capacity at it, but turns out that never works in a panicked rush either. Also, our system designed to handle outage notifications wasn't capacity planned, like, at all.
I know I have had my share of server issues, but it seems to me that many 'cloud' services out there are simply adding too many layers of abstraction that tend to make things very, very touchy to any small issue occurring. Because of this I try to keep my server stacks/frameworks as basic as possible while still implementing performance oriented services like NoSQL, caching, etc.
"...streaming data API which connects the dyno manifold to the routing mesh."
Give me a break!
1) to prevent the average, non-technical person from understanding it ("Phew, I'm glad these guys are figuring out this stuff and not me - that's why I host with them. I don't even know what a 'dyno manifold' is!")
2) to show management how smart we are and that you still need us ("because who else is going to figure this 'routing mesh' stuff out if you fire those responsible for the outage")
A simple "we're sorry and we've given 10 lashes to the engineer performing the manual garbage collection" may have been a better approach.
Having said that, I still think Heroku is awesome.
The manual garbage collection wasn't the problem. An unexpected data structure created by garbage collection wasn't handled in a fault tolerant manner.
Instead of "dyno", they could possibly use a word like "VM". Except that they're not really virtual machines, nor are they EC2 instances. Read Only Chroot Jails plus Precompiled Application, Libraries, and Environment (ROCJPALEs?) They also have a pretty complex set of support structures that provide connectivity to databases and other resources. Perhaps someone can suggest an existing name for that, but I know of none.
Instead of "manifold", perhaps they could use the word "cluster". Except it's not really a cluster, it's a set of distributed clusters. And nodes in a cluster are typically machines. The nodes in the dyno manifold aren't machines, virtual machines, they're ROCJPALEs. You could use the word "array", but again, it's not really an array. It's a multi-layered, geographically distributed structure of co-hosted application jails. "Manifold" seems as good a term as any.
"Streaming" seems like a good word. It's specifically relevant to this incident... they describe how the API is not atomic; that each message is built on top of the previous entries, and the data structures are implicit in the stream. That sounds like the definition of "streaming" to me.
"API" seems like a widely accepted term. They could've described it as a "protocol", perhaps. But neither seems more jargony than the other.
"Data"... well I suppose "streaming API" without the data would work. But it serves to differentiate it from a streaming video protocol.
"Mesh" has a very specific meaning. It means that you have a set of nodes that are connected peer-to-peer and that messages travel through the network by hopping from node to node. I'm assuming that their routing layer is organized in this way.
"Routing" is also pretty well defined. Requests come in and need to be sent to the machine that can serve responses to it. What would you call that instead of routing?
I feel like people who object to this kind of language are the same folks who object to the word "cloud". People don't take the time to understand different strategies to provisioning and application hosting APIs, and then think these words don't mean anything. Yeah, salespeople use the word to hustle the Same Old Shit, but it also actually means something to people like us who are actually building stuff.
We have seen very elaborate post-mortems from google, facebook, twitter, and no least from Amazon themselves (you know, the playground that heroku builds their sandcastles in).
The aforementioned companies had no problem explaining their respective issues in plain language that every engineer did understand.
Heroku doesn't even try to explain themselves. They just throw around fantasy words without real explanations, seemingly overwhelmed by their own awesomeness (in a failure report, no less).
As an engineer I feel insulted by this pamphlet. All I can gather from it is that they screwed up and apparently somehow related to their request-routing layer. Thanks, we knew as much before reading that text.
I still have no idea what actually went wrong and how they intend to prevent it in the future. But I'll certainly advise people to avoid a company that babbles about "control rods" when their software screws up.
https://devcenter.heroku.com/articles/dynos
It explains all :-)
This seems to be an unfortunate attempt to apply Corporate Speak to a technical announcement; "Let's see how many paragraphs we can fill with technical-sounding gibberish without actually telling anything..."
If they're too embarrassed to tell what happened then they should just keep quiet. Don't insult your customers with handwavy bullshit bingo, that just leaves a sour taste in everyones mouth.
Just imagine the hilarity when the PHB asks his inhouse engineer to translate this "post-mortem" into layman's terms for him. Most bosses have a bit of humor, but not when it comes to hosting infrastructure.
This is techno-babble on a scale the world has never seen!
We're expecting them to be the A-Grade tech wizards who can give us 0 down time. They are after expecting thousands of people to trust their services and to outsource the server hosting and administration duties to them.
So they tread the fine line between convenience (and related "cloud" benefits) and "I can do this myself".
If they can't give us the assurances that they can do it better, cheaper and more reliably than we can do it ourselves then what good are they?
If they can't capacity plan a simple System Status page (running on Rackspace) and keep that up and running then what good are they?
And since their service appeals to a certain level of geek competence, they also can't get away with techno babble bull shit responses to outages.
Exactly. But something like this which they said makes them seem so ordinary:
"The improved status site allows users to subscribe to notifications when an incident is opened. As a result, our status site experienced unprecedented spikes of load during this incident. This high load crushed the site,"
Basically saying whatever they setup for a status site choked on sending out emails or sms, as if they were hosted on a shared server and got mentioned simultaneously on a few major sites.
The thundering herd of recovery is especially difficult to cope with: your error recovery code can work just fine for normal outages but then fail completely when faced with just a few more components going dark.
I'm going to assume their routing mesh has many points of ingress and a larger number of exit paths (the dyno manifold), but that the nodes they've got participating in the mesh are actually in some sort of mesh topology (or form a connected graph).
This has the upside that if you lose several nodes in the mesh you probably haven't lost a path to any dynos. If you lose a whole AZ you can spin up new dynos in one of the existing ones and reconfigure the mesh quickly. My experience with loadbalancers, especially big load balancers is that updating a large swath of VIPs is NOT a fast operation (although you would start failing health checks on the missing nodes pretty quickly, adding new capacity to replace them is hard).
The mesh has the downside that the failure modes are a lot more complicated. Oh, and nobody knows what the hell you're talking about.
Of course, I could be wrong. They could just be using NetScalers (or ELB) and calling it a "routing mesh".
Yes. It's a level of detail that borders on obfuscation.
fwiw, I've always used the term "load balancer" to also refer to two redundant load balancing machines. (If I worked with more complex load balancers, I doubt I'd stop.) In the general sense, it just means "the apparatus that balances the load".
Routing mesh? We call that a cluster of load balancers in the real world.
Things like EC2 RDS AWS S3 EBS EMR IAM AMI SQS SNS SES HPC VPC
But Amazon is the reference in cloud hosting and these terms are well understood in the field.
Heroku also had to coin some words to describe their architecture. But frankly this outage report is worthy of an Hollywood hacker movie:
"A manual garbage collection process which created an unusual record in the data stream" Wow!