Cloudflare Down Again – and DownDetector Is Also Down https://downdetector.tr/ |
Cloudflare Down Again – and DownDetector Is Also Down https://downdetector.tr/ |
If this is unwrap() again, we need to have a talk about Rust panic safety.
And I'm 100% sure the management responsible for this is already fueling up the ferraris to drive to their beach house. All of us make them rich and they keep on enshittifying their product out of pure hubris.
I have stopped fighting this battle at work (not Cloudflare). Despite Friday being one of the most important days of the week for our customers, people still deploy the latest commit 10 minutes before they leave in the afternoon. Going on a weekend trip home to your family? No problem, just deploy and be offline for hours while you are traveling...
The response was that my way of thinking is "old school". Modern development is "fail fast" and that CI/CD with good tests and rollback fixes everything. Being afraid of deploys is "so last decade"... The problem is that our tests don't cover everything, it may not fail fast, and not all deploys can be rolled back quickly and the person who knows what their huge commit that touches multiple files actually does is unavailable!
We have had multiple issues with late afternoon deploys, but somehow we keep doing this. Funnily enough, I have noticed a pattern. Most experienced devs stops doing this after causing a couple of major downtimes, due to the massive backlash from customers while they are scrambling to fix the bug. So gradually they learn to deploy at less busy times and monitor the logs to be able to catch potential bugs early.
The problem is that not enough has learned this lesson because they have been lucky that their bugs have not been critical, or because they are too invested in their point of view to change. It seems that some individuals learn the hard way, but the organization has not learned or is reluctant to push for a change due to office politics. I decided to keep my head low and let things play out, as I simply no longer care as long as the management don't care either.
- Friday
- Christmas time
- Affecting both shopify.com and claude.ai, so no phased deployment
- Takes 30 minutes to remediate
If they would've just deployed to a single of their high-value customers at once, they could've spared shopify.com an hour of downtime and maybe millions in abandoned shopping carts.
In fact, there are incentives for public failures: they'll help the politicians that you bought sell the legislation that you wrote explaining how national security requires that the taxpayer write a check to your stockholders/owners in return for nothing.
"A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components."
The bug is known since several days, and the hotfix was already in place. So they worked on the "final fix" and chose to deploy it on a friday morning.
Extrapolating at current rates I guess that means April 2026.
Basically, my take is: It’s not a technical monoculture; it’s a billing psychology + inertia culture.
I dont think the internet is fragile simply because Cloudflare is so ubiquitous, because that view ignores the economic factor of why people choose them. The situation is really a perfect bi-modal distribution: at the low end, you have hobbyists and personal sites who use Cloudflare because it is the only viable free option, and at the extreme high end, you have massive enterprises that truly need that specific global capacity to scrub terabits of attack traffic.
However, I think the following perspective is important: For the vast middle ground of the internet—most standard businesses and SaaS platforms—Cloudflare could be viewed as redundant. If you are hosting on AWS, Google Cloud, or Azure, you are already sitting behind world-class infrastructure protection that rivals anything Cloudflare offers. The reason this feels like a dangerous monoculture isn't because Google or Amazon can't protect you, but rather because Cloudflare wins on the psychology of billing. They sell a flat-rate insurance policy against attacks, whereas the cloud giants charge for usage, which scares people.
Ultimately, the internet isn't suffering from a lack of technical alternatives to DDoS protection, nor is Cloudflare a NECESSARY single point of failure; it is just suffering from a market preference for predictable invoices over technical redundancy, and inertia, leading to an extremely high usage of Cloudflare. So basically: Even though we are currently relying a lot on Cloudflare, we are far from vendor lock-in, and there is a clear path to live without them, given that there are many alternatives.
Maybe we could view this as a good thing, since basically medium to large-scale enterprises efficiently subsidize small and hobby-level actors? So to summerize: The 2018-era "just use Cloudflare for everything" advice is outdated, and the following is a better philosopy: If you're tiny: Cloudflare free tier is still a no-brainer. If you're huge and actually get attacked: pay for Cloudflare Enterprise or equivalent.
If you're anywhere in between: seriously consider whether you need it at all. The hyperscalers are good enough, and removing Cloudflare can actually improve your availability (fewer moving parts).
I think Cloudflare thinks this way too, which is why they've been pushing Zero Trust, Workers, WARP, Access, and Magic Transit, to become the default network stack for companies, not just the default firewall.
/wall-of-text
Each downed page diminishes me,
for I am involved in WANkind.
Therefore, ping not to know
for whom the downdetector detects,
it detects for thee.Look, i think we need a resilient system that routes packets via multiple possible pathways, preferably all of them, so that ideally nothing is ever fully down. We can name that system the undownnet
1. There were outages under the old guard.
2. The new guard is operating systems that are larger than what the old guard operated.
Some interesting DNS data https://news.ycombinator.com/item?id=46159249
That blog post made it to the front page of HN and my site did not go down. Nor did any DDoS network take the site out even though I also challenged them last time by commenting that I would be okay with a DDoS. I would figure out a way around it.
In general, marketing often works via fear, that's why Cloudflare has those blog posts talking about "largest botnet ever". Advertisement for medicine for example also works often via fear. "Take this or you die", essentially.
Stop trying to devalue labor. Not much sympathy when you’re obviously cutting corners.
Obviously, 'The Crying Boy' was not the cause of the fires, it was just that most homes in the 80s England had that print, as it was a popular one, and people found a pattern where there wasn't one.
These companies also don't vibe code (which would involve just prompting without editing code yourself, at least that's the most common definition).
I really hope news like these won't be followed by comments like these (not criticism of you personally) until the AI hype dies down a bit. It's getting really tiresome to always read the same oversimplified takes every time there's some outage involving centralized entities such as cloudflare instead of talking about the elephant in the room, which is their attempt of doing MITM on the majority of internet users.
downdetectorsdowndetectorsdowndetector.com and downdetectorsdowndetectorsdowndetectorsdowndetector.com seem like they might be legit. One has the results in the HTML, the other fetches some JSON from a backend (`status4.php`).
Would you want me to:
- Create a list of all LLM models released in the past few months
- Let you know why my existence means you can't afford RAM anymore
- Help you learn sustenance farming so that you can feed your family in the coming AI future?
I'm on the pro plan, only using Sonnet and Haiku. I almost never hit the 5-hour limit, let alone in less than 2 hours.
No cloudflare no problem
At what point does the cost outweigh the benefit?
Well, one way is to use a different DNS provider than either of your hosting options.
You can see this is getting complicated. Might be better to take the downtime.
But if I had to make a real recommendation I’m not aware of any time in the last decade that a static site deployed on AWS S3/Cloudfront would have actually been unavailable.
You list multiple nameservers.
yoursite.com. 86400 IN NS ns1.yourprovider.com.
yoursite.com. 86400 IN NS ns2.yourprovider.com.
yoursite.com. 86400 IN NS ns1.yourotherprovider.com.
yoursite.com. 86400 IN NS ns2.yourotherprovider.com.They haven't had an incident that bad since they switched from C to Rust.
Excel crashed? Must be that new WiFi they installed!
They expected a maximum config size but an upstream error meant it was much larger than normal. Their Rust code parsed a fraction of the config, then did ".unwrap()" and panicked, crashing the entire program.
This validated a number of things that programmers say in response to Rust advocates who relentlessly badger people in pursuit of mindshare and adoption:
* memory errors are not the only category of errors, or security flaws. A language claiming magic bullets for one thing might be nonetheless be worse at another thing.
* there is no guarantee that if you write in <latest hyped language> your code will have fewer errors. If anything, you'll add new errors during the rewrite
* Rust has footguns like any other language. If it gains common adoption, there will be doofus programmers using it too, just like the other languages. What will the errors of Rust doofuses look like, compared to C, C++, C#, Java, JavaScript, Python, Ruby, etc. doofuses?
* availability is orthagonal to security. While there is a huge interest in remaining secure, if you design for "and it remains secure because it stops as soon as there's an error", have you considered what negative effects a widespread outage would cause?
The last outage was in fact partially due to a Rust panic because of some sloppy code.
Yes, these complex systems are way more complex than just which language they use. But Cloudflare is the one who made the oversimplified claim that using Rust would necessarily make their systems better. It’s not so simple.
They might go on a hiring freeze, cancel a role, or in some cases pass on someone asking too much... But I don't think any major players are actively out trawling for "cheap and dumb". Certainly not Cloudflare, AWS and Google.
Rust did its job and forced them to return an error from the lower function. They explicitly called a function to crash if that returned an error.
That’s not a rust problem.
What we do know is Cloudflare wrote a new program in Rust, and never tested their Rust program with too many config items.
You can't say "Rust did its job" and blame the programmer, any more than I can say "C did its job" when a programmer tells it to write to the 257th index of a 256 byte array, or "Java did its job" when some deeply buried function throws a RuntimeException, or "Python did its job" when it crashes a service that has been running for years because for the first time someone created a file whose name wasn't valid UTF-8.
Footguns are universal. Every language has them, including Rust.
You have to own the total solution, no matter which language you pick. Switching languages does not absolve you of this. TANSTAAFL.
You absolutely can. This is someone just calling panic in an error branch. Rust didn’t overrun the memory which would have been a real possibility here in C.
The whole point is that C could have failed in the exact same way but it would have taken extra effort to even get it to detect the issue an exit. For an error the programmer didn’t intend to handle like in this case, it likely would have just segfaulted because they wouldn’t bother to bounds check.
> TANSTAAFL
The way C could have failed here is a superset of how Rust would. Rust absolutely gives you free lunch, you just have to eat it.
What other popular site has zero images or video to speak of?
Also cloudflare is needed due to DDOS and abuse from rogue actors, which are mostly located in specific areas. Residential IP ranges in democratic countries are not causing the issues.
If you work harder at taking the burden upon yourself to understand others, you might be surprised how well people can learn to communicate despite differing backgrounds.
I'm not saying I always understand 100% of what is said. When someone with an accent from a specific part of a country speaks super fast and is on a poor line with lots of street traffic in the background, it can be hard to follow. But usually I catch enough of it to be able to communicate.
Only once have I encountered a problem. A colleague berated me in front of others for speaking "difficult English" and accused me of doing this on purpose to cause trouble for them, instead of speaking proper international English like everyone else did. But, I am a native English speaker with an RP accent and we were all in England at the time, working for a British organisation. I was simply speaking normally and otherwise had no issue with this colleague, whose English was very good. I don't recall their having been any misunderstandings between us before.
The main bad traffic that I receive comes from server IP ranges all over the world and several rogue countries who think it makes sense to wage hybrid war against us. But residential IP ranges are not the majority of bad traffic.
I would even say that residential IP ranges are most of the paying customers for companies, and if you just block everything else you most likely wouldn't need to use cloudflare.
Unfortunately firewall technology is not there yet. It's quite hard to block entire countries, even harder to block any non-residential ASN. And then you can still add some open source "i am human" captcha solution before you need to use cloudflare.