When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug(blog.cloudflare.com) |
When “idle” isn't idle: how a Linux kernel optimization became a QUIC bug(blog.cloudflare.com) |
This one just feels... off. The buildup just doesn't feel right.
The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates.
And once I finally made it to the end I read the following:
> If you're interested in congestion control, transport protocols, or contributing to open-source networking code, check out the quiche repository. We're always looking for talented engineers who love digging into problems like these, please explore our open positions.
You don't add that to your blog-posts 5 days after laying off 20% of the company, regardless of whether they're sales people or engineers. If you want to add it, delay the post by 2 weeks.
Equally, there is only 1 role open in Engineering, and it's an intern role, posted yesterday:
https://www.cloudflare.com/careers/ (filter by Engineering).
Did they lay off their PR team as well?
You're being too kind -- "The [b]epoch[b] is the reference timestamp CUBIC". Weird style to have random bold words. Its a blog post for the sake of it - no real takeaway. Well, there is the takeaway section that's a special summary of the article, instead.
I found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and especially in datacenters with large pipes, seems so slooow to me. Almost two seconds with no loss whatsoever till achieving BDP again and then shooting itself in the foot every time it hits the ceiling. Each one of those losses a retransmission.
This is somewhat funny to read because this specific issue in CUBIC (sudden CWND jump upon existing quiescence) was originally discovered in Google's QUIC library and then later reported to the team working on the TCP stack. I know this because I was the one who found that bug back in 2015.
That said, congestion control algorithms are really prone to logic bugs, and very subtle changes in the algorithm can often lead to dramatically different outcomes. Because of that, there's a lot of value in running congestion control code that has been tested on a wide variety of real Internet traffic.
As far as I know, while they might have either way, they did not ("rewrite QUICK [...] for use in userspace"): the linux kernel implementation only landed late 2025. Quiche was started ca 2018 (that's when Cloudflare started beta-deploying QUIC, the first public alpha of quiche was january 2019).
I don't know that there even was an in-kernel implementation of quic before msquic.sys which I believe first shipped in Server 2022 circa mid 2021 (and is used as the implementation backend by MsQuic on Server 2022 and W11).
Imagine your team does not want to write a test because it's too much work or hard to model - this is a great example to bring up.
What is a CCA in this context?
After the initial packet loss triggered purposefully the first two seconds in this experiment, the only thing which could cause loss is the network queue (i.e. a simple tail drop, fq-codel, etc) which cannot process packets faster than they can arrive. At this point the link is saturated. The loss becomes a signal for CUBIC to reduce its window. This causes the oscillations you pointed out.
Unlike CUBIC, BBR [0] uses a model-based approach that estimates the available bandwidth and leaves some headroom kind of like you suggest to achieve higher throughput, and doesn't react as aggressively to loss as CUBIC.
[0] https://datatracker.ietf.org/meeting/104/materials/slides-10...
May not be related, but it might. It only recently started happening to me/our sites in the past couple of days.
Most control loops have the same property. The path the system follows 99% of the time gets well-exercised; the path it falls into when things go wrong is the path you actually need to be correct on. There's usually no way to discover the bug until you deliberately drive the system into the bad regime and watch it try to climb out.
The LLM pass (unsurprisingly) made it worse.
For example:
The results were conclusive: 100% pass rate, showing Reno recovered cleanly after the loss phase, and revealing that this is a CUBIC-related bug.
Look, I’m reading a description of a Linux kernel network congestion bug. I don’t need the hand-holding.
I wonder what happened with the very hardcore engineering that used to happen at Cloudflare and was published? Almost every blog post today seems to expose some weirdness at Cloudflare, rather than highlighting excellence in engineering, what changes? Been slowly changing over the years, did they change their hiring practices or something?
I don't understand where or how AI picked up that habit, because it's self evidently terrible. It makes it clear how low signal AI based writing is. The writing is like the music in shitty blockbusters; engineered to make you feel, rather than to actually structure the content or provide meaningful sections.
Compare this writeup to the Pixter writeup, where sections feel natural and not "scripted" like this.
https://doi.org/10.1145/3452296.3472912
Toward formally verifying congestion control behavior | Proceedings of the 2021 ACM SIGCOMM 2021 Conference
So instead of:
> President Jackson said "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.".
They'll do something like:
> President Jackson noted that "Lorem ipsum dolor sit amet". The head of state also remarked that "consectetur adipiscing elit" while emphasizing that "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua".
> "Ut enim ad minim veniam, quis nostrud exercitation", categorically proclaimed the former business tycoon. He concluded that "ullamco laboris nisi ut aliquip ex ea commodo consequat".
I've seen this way before LLMs and how much it's used varies a bit from language to language. But it's so formulaic, I can't help but imagine some brain-dead moron sitting in front of the keyboard, trying to make 5 paragraphs from 2 sentences someone said without adding anything else.