Details of yesterday's Bunny CDN outage(bunny.net) |
Details of yesterday's Bunny CDN outage(bunny.net) |
I wonder, if simple checksum verification of the file would have helped in avoiding this outage all together.
> Turns out, the corrupted file caused the BinaryPack serialization library to immediately execute itself with a stack overflow exception, bypassing any exception handling and just exiting the process. Within minutes, our global DNS server fleet of close to a 100 servers was practically dead
This is exactly, why one needs a canary based deployments. I have seen umpteen amounts of issues being caught in canary, which has saved my team tons of firefighting time.
Oh man, you stirred up a really old Cloudflare memory. Back when I was working on our DNS infrastructure I wrote up a task that says: "RRDNS has no way of knowing how many lines to expect or whether what it is read is valid. This could create an issue where the LB map data is not available inside RRDNS."
At the time this "LB map" thing was critical to the mapping between a domain name and its associated IP address(es). Without it Cloudflare wouldn't work. Re-reading the years old Jira I see myself and Lee Holloway discussing the checksumming of the data. He implemented the writing of the checksum and I implemented the read and check.
I miss Lee.
I'm deeply moved after reading it. Can't imagine how tragic it must be for people who know Lee.
For serialization in a distributed system you always want to have a parser which can detect invalid data and has means to support forward and backward compatibility.
Also for HTTP requests I suppose
Your system WILL go down eventually. The question is how will you recover from it?
We are dropping BinaryPack mainly because we're a small team, and it wasn't really a big benefit anyway, so spending more time than necessary to try and salvage that makes no sense. This was more of a hot-fix since we don't want the same thing repeating in a week.
But also they used their own infra it feels to setup their stuff and if their infra was dead they couldnt rollback, which sounds like a case where people a bit too optimistic.
We had catastrophes too, notably on poison pills in a record stream we cant alter, but this update cascade crash sounds avoidable.
Always easy to judge anyway, always happens to you eventually :D
I'm sure I'd be fuming if I worked at some multi-million dollar company but as someone that mainly works for smaller businesses it's not the end of the world, I suspect most of my clients haven't even noticed yet.
Anyway, it’s always DNS. Always.
“Unfortunately, that allowed something as simple as a corrupted file to crash down multiple layers of redundancy with no real way of bringing things back up.”
You can spend many, many millions of $ on multi-AZ Kubernetes microservices blah blah blah and it’ll still be taken down by a SPOF, which, 99% of the time, is DNS.
Actual redundancy, as opposed to “redundancy”, is extremely difficult to achieve because the incremental costs of one more 9 are almost exponential.
And then a customer updates their configuration and your entire global service goes down for hours ala Fastly.
Or a single corrupt file crashes your entire service.
Which is disappointing. An infrastructure where the backend is VERY easy to make highly redundant. Thwarted by decisions not to do that easy work, or thwarted by client libraries that don't take advantage of it.
I assume that they are using .Net, as SOEs bring down .Net processes. While that sounds like a strange implementation detail, the philosophy of the .Net team has always been "how do you reasonably recover from an stack overflow?" Even in C++ what happens if, for example, the allocator experiences a stack overflow while deallocating some RAII resource, or a finally block calls a function and allocates stack space, or... you get the idea.
The obvious thing to do here would be to limit recursion in the library (which amounts to safe recursion usage). BinaryPack does not have a recursion limit option, which makes it unsafe for any untrusted data (and that can include data that you produce, as Bunny experienced). Time to open a PR, I guess.
This applies to JSON, too. I would suggest that OP configure their serializer with a limit:
Bunny has a great interface and service; I'm really surprised how little people know about it, I think I discovered it on some 'top 10 CDNs list' that I usually ignore, but the pricing was too good to pass up.
The team is really on the ball from what I've seen. Appreciate the descriptive post, folks!
So to add some details, we already use multiple deployment groups (one for each DNS cluster). We always deploy each cluster separately to make sure we're not doing something destructive. Unfortunately this deployment went to a system that we believed was not a critical part of infrastructure (oh look how wrong we were) and was not made redundant, since the rest of the code was supposed to handle it gracefully in case this whole system was offline or broken.
It was not my intention to blame the library, obviously this was our own fault, but I must admit we did not expect a stack overflow out of it, which completely obliterated all of the servers immediately when the "non-critical" component got corrupted.
This piece of data is highly dynamic and processes every 30 seconds or so based on hundreds of thousands of metrics. Running a checksum did nothing good here, because the distributed file was perfectly fine. The issue happened when it was being generated, not distributed.
Now for the DNS itself, which is a critical part of our infrastructure.
We of course operate a staging environment with both automated testing and manual testing before things go live.
We also operate multiple deployment groups so separate clusters are deployed first, before others go live, so we can catch issues.
We do the same for the CDN and always use canary testing if possible. We unfortunately never assumed this piece of software could cause all the DNS servers to stack overflow.
Obviously, I mentioned, we are not perfect, but we are trying to improve on what happened. The biggest flaws we discovered were the reliance on our own infrastructure to handle our own infrastructure deployments.
We have code versioning and CI in place as well as the options to do rollbacks as needed. If the issue happened under normal circumstances, we would have the ability to roll back all the software instantly, and maybe experience a 2-5 minute downtime. Instead, we brought down the whole system like dominos because it all relied on each other.
Migrating deployment services to third-party solutions is therefore our biggest fix at this point.
The reason we are moving away from BinaryPack is because it simply wasn't really providing that much benefit. It was helpful, but it wasn't having a significant impact on the overall behavior, so we would rather stick with something that worked fine for years without issues. As a small team, we don't have the time or resources to spend improving it at this point.
I'm somewhat exhausted after yesterday, so I hope this is not super unstructured, but I hope that answers some questions and doesn't create more of them :)
If I missed any suggestions or something that was unclear, please let me know. We're actively trying to improve all the processes to avoid similar situations in the future.
We just wrapped up the first season, but I'm going to put this on the list of episodes for the second season: https://downtimeproject.com.
Nice work Bunny CDN.
Very similar to the story about the planes with holes coming back in WWII and the initial analysis of adding more armor to where the holes were, when someone flipped it and pointed out that armor was needed where the holes weren't since planes with holes in those spots weren't the ones coming back.
There is a no way it’s DNS
It was DNS
One of the most bittersweet haikus for any sysadmin :p
> To do this, the first and smallest step will be to phase out the BinaryPack library and make sure we run a more extensive testing on any third-party libraries we work with in the future.
Sure. Not exactly a structural fix. But maybe worth doing. Another view would be that you've just "paid" a ton to find issues in the BinaryPack library, and maybe should continue to invest in it.
Also, "do more tests" isn't a follow up. What's your process for testing these external libs, if you're making this a core part of your reliability effort?
> We are currently planning a complete migration of our internal APIs to a third-party independent service. This means if their system goes down, we lose the ability to do updates, but if our system goes down, we will have the ability to react quickly and reliably without being caught in a loop of collapsing infrastructure.
Ok, now tell me how you're going to test it. Changing architectures is fine, but until you're running drills of core services going down, you don't actually know you've mitigated the "loop of collapsing infrastructure" issue.
> Finally, we are making the DNS system itself run a local copy of all backup data with automatic failure detection. This way we can add yet another layer of redundancy and make sure that no matter what happens, systems within bunny.net remain as independent from each other as possible and prevent a ripple effect when something goes wrong.
Additional redundancy isn't a great way of mitigating issues caused by a change being deployed. Being 10x redundant usually adds quite a lot of complexity, provides less safety than it seems (again, do you have a plan to regularly test that this failover mode is working?) and can be less effective than preventing issues getting to prod.
What would be nice to see if a full review of the detection, escalation, remediation and prevention for this incident.
More specifically, the triggering event here, the release of a new version of software, isn't super novel. More discussion of follow ups that are systematic improvements to the release process would be useful. Some options:
- Replay tests to detect issues before landing changes
- Canaries to detect issues before pushing to prod
- Gradual deployments to detect issues before they hit 100%
- Even better, isolated gradual deployments (i.e. deploy region by region, zone by zone) to mitigate the risk of issues spreading between regions.
Beyond that, start thinking about all the changing components of your product, and their lifecycle. It sounds like here some data file got screwed up as it was changed. Do you stage those changes to your data files? Can you isolate regional deployments entirely, and control the rollout of new versions of this data file on a regional basis? Can you do the same for all other changes in your system?
> One thing you could do in future is to url redirect any BunnyCDN url back to the clients original url, in essence disabling the CDN and getting your clients own hosts do what they were doing before they connected to BunnyCDN, yes it means our sites won't be as fast but its better than not loading up the files at all. I wonder if that is possible in technical terms?
Isn't this a horrible idea? If you use bunny, this would cause a major spike in the traffic and thus costs from your origin server.
This is also a terrible idea because of ACLs/firewalls only allowing traffic from CDN (this is extremely common for things like Cloudflare and Akamai) and relying on the CDN for access control.
There's a reason I use a CDN, let me decide if it's up or down if the CDN is down. If I want failover, I'll do that bit.
My take aways though were they should have tested the update better. They should have their production environment more segmented with staggered updates so they have much more contained disasters. And they should have had much better catastrophic failure plans in place.
It tought me a valuable lesson: make sure it is easy to switch to another CDN & to update cached/stored urls
Frankfurt was mentioned in the post and I immediately thought it would be a bad idea because I’ve always seen USA to Germany traffic have higher latency. Maybe within Europe it’s fine.
> Both SmartEdge and the deployment systems we use rely on Edge Storage and Bunny CDN to distribute data to the actual DNS servers. On the other hand, we just wiped out most of our global CDN capacity.
That’s the TLDR. What a stressful couple of hours that must have been for their team.
That's around 4:25 a.m EST. Are updates usually done around this time at other companies? Seems like that's cutting pretty close to the around 8am where a lot of employees start working.
The details of the whole incident sounds pretty terrifying and I am inspired to hear how much pressure their admins were under and got it working again. Good work.
Seems like they were updating production during work hours for most people which is pretty odd imo. Usually I would expect them to get this done between midnight and 2-3am.
The feature set is pretty full, no edge functions, but there is a rule engine you can run on the edge. Fast config updates, nice console and works well enough for most of my projects.
They also have a nice integrated storage solution that's way easier to configure than S3 + Cloudfront, and lots of origin shielding options.
My primary use has been for serving image assets, switched over from Cloudfront and have seen probably a >80% cost reduction, and no noticeable performance reduction, but as I mentioned I'm operating at a scale where milliseconds of difference don't mean much.
You can see our CDN usage inspecting the URLs to the product images. Size attributes are added to the URL and Bunny automatically resizes and compresses the images on the fly.
AWS Cloudfront is neither the fastest or cheapest, in both bulk and small file transfer. I cant think of a single technical reason it is better. Fastly, Akamai, Limelight, Cloudflare or even good old EdgeCast. They all have their strong point in some of their niche services or domain.
Any reason for using AWS Cloudfront Enterprise purchase reason? Or are there some technical superiority I am not seeing?
The most effective tools for the job are usually the more dangerous ones. Certainly, you can do anything without recursion, but forcing this makes a lot of problems much harder than they need to be.
Can you expand on this or link to any further reading? I just realized that this affects my platform (Go) as well, but I don't understand the reasoning. Why can't stack overflow be treated just like any other exception, unwinding the stack up to the nearest frame that has catch/recover in place (if any)?
Consider the following code:
func overflows() {
defer a()
fmt.Println("hello") // <-- stack overflow occurs within
}
func a() {
fmt.Println("hello")
}
In truth, the problem exists because of implicit calls at the end of methods interacting with stack overflows, whether that's because of defer-like functionality, structured exception handling, or deconstructors.
I had a "high school" level programming class with Python before studying CS. I ran into CPython's recursion limit often and wondered why one would use recursion when for loops were a more reliable solution.
Nowadays, my "recursion" is for-looping over an object's children and calling some function on each child object.
Perhaps you guys need a ... database? Relevant HN discussion few months ago when tailscale migrated from json to etcd: https://news.ycombinator.com/item?id=25767128
This read like the DNS software is being changed. As some people already mentioned is this a corruption where checksum would of been prevented the stack overflow or would a canary detected this? Why would a change to DNS server software not canaried?
GP also notes that this database changed perhaps every 30 seconds
Just a few guesses.. if you have a process that corrupts a random byte every 100.000 runs, and you run it every 30 seconds, it might take days before you're at 50% odds of having seen it happening. and if that used to be a text or JSON database, flipping a random bit might not even corrupt anything important. Or if the code swallows the exception at some level, it might even self-heal after 30 seconds when new data comes in, causing an unnoticed blib in the monitoring if at all
Now I don't know what binary pack does exactly, but if you were to replace the above process with something that compresses data, a flipped bit will corrupt a lot more data, often everything from that point forwards (where text or json is pretty self-syncronizing). And if your new code falls over completely if that happens, no more self-healing.
I can totally imagine missing an event like that during canary testing
Deploying to a different, lower priority system is not a canary. Do you phase deployments to each system, per host or zone?
looks like storage zones are still not fully stable ? after experiencing several issues with storage zones earlier, we migrated to pull a zone. we didn't had any major issues after the migration.
what plans do you have to improve your storage zone ?
Also, their DNS broke last month as well, but I guess we won't mention that as it would invalidate 2 years of stellar reliability
I don't know the details of your operation, but keeping your ability to update your systems separate from your systems is something I'd strongly encourage.
https://en.wikipedia.org/wiki/Friedrich_Nietzsche#Mental_ill...
Perhaps one difference is that, while panics are always avoidable in a recovery function, stack overflows are not (if it happens to be deep enough already). Does the argument go “even a seemingly safe recovery function can’t be guaranteed to succeed, so prevent the illusion of safety”?
(To be clear: I’m not arguing, just trying to understand.)
In the absolute worst case scenario: stack unwinding is itself a piece of code[1]. In order to initiate the stack unwind, and deal with SEH/defer/dealloc, the Go runtime would need stack space to call that method. Someone might say, "freeze the stack and do the unwind on a different thread." The problem is the bit in the quotes is, again, at least one stack frame and needs stack space to execute.
I just checked the Go source, and it basically uses a linked list of stack frames in the heap[2]. If a stack is about to overflow, it allocates a new stack and continues in that stack. This does have a very minor performance penalty. So you're safe from this edge case :).
[1]: https://www.nongnu.org/libunwind/ [2]: https://golang.org/src/runtime/stack.go
To this day US PoPs are still pulling from EU storage servers (our storage zone is in NY, replicated in DE).
< Server: BunnyCDN-IL1-718
< CDN-RequestCountryCode: US
< CDN-EdgeStorageId: 617
< CDN-StorageServer: DE-51
Would love additional capabilities within the image optimizer such as methods of crop