The primary tradeoff of initcwnd is setting a reasonable window before you've learned anything about the path. BBR has little say on this because it takes, in relative terms, quite a while to go through its phases. An early BBR session is therefore not really superior to other congestion controls because that is not the problem it is really focused on.
Jacking up the initcwnd, you start to risk tail loss, which is the worst kind of loss for a sliding window.. especially in the primordial connection. There are ways of trying to deal with all that but they are loss predictions.
If you are a big enough operator, maybe you have some a priori knowledge to jack this up for certain situations. But people are also reckless and do not understand the tradeoffs or overall fairness that the transport community tries to achieve.
As other comments have pointed out, QUIC stacks also replicate congestion control and other algorithms based on the TCP RFCs. These are usually much simpler and lacking features compared to the mainline Linux TCP stack. It's not a free lunch and doesn't obviate the problem space any transport protocol has to make tradeoffs on.
Having to pick a particular initcwnd to be used for every new TCP connection is an architectural limitation. If they could collect data about each destination and start each TCP connection with a congestion window based on the recent history of transfers from any of their servers to that destination, it could be much better.
It's not a trivial problem to collect bandwidth and buffer size estimates and provide them to every server without delaying the connection, but it would be fun to build such a system.
Tons of fun. Sadly, I don't have access to enough clients to do it anymore.
But napkin architecture. Collect per connection stats and report on connection close (you can do a lot with tcp_info or something something quic). That goes into some big map/reduce whatever data pipeline.
The pipeline ends up with some recommended initial segment limit and a mss suggestion [1], you can probably fit both of those into 8-bits. For ipv4, you could probably just put them into a 16 MB lookup table... shift off the last octet of the address and that's your index into the table. For ipv6 it's trickier, the address space is too big; there's techniques though.
At google scale, they can probably regenerate this data hourly, but weekly would probably be plenty fast.
[1] This is it's own rant (and hopefully it's outdated) but mss on a syn+ack should really start at the lower of what you can accept and what the client told you they can. Instead, consensus has been to always send what you can accept. But path mtu doesn't always work, so a lot of services just send a reduced mss. If you have the infrastructure, it's actually pretty easy to tell if clients can send you full mtu packets or not... with per network data, you could have 4 reasonable options, reflect sender, reflect sender minus 8 (pppoe), reflect sender minus 20 (ipip tunnel), reflect sender minus 28 (ipip tunnel and pppoe). If you have no data for a network, random select.
They also miss the fact that even with an initcwnd of 10 the TLS negotiation isn’t going to consume it so the window starts growing long before content is actually sent.
Plus there’s no discussion of things like packet pacing
I don't think that's true. QUIC implementations typically use the same congestion control algorithms, including both CUBIC and BBR, at least nominally. The latest RFCs for those discuss use with both TCP and QUIC. Though, perhaps when used with QUIC they have more degrees of freedom to tune things.
A somewhat-interesting, though simplistic, discussion of congestion control in TCP.
And then… "but don't worry, QUIC is magic, and it doesn't need any of this.”
No. No it's not. QUIC is not magic and it basically changes nothing in terms of congestion control.
Basically it sqrt's your actual packet loss rate as far as feedback frequency/density is concerned, without actually even typically having to enact that loss. For example, you could get congestion feedback every 100th packet (as you would with 1% packet loss), during network conditions that would traditionally only have 0.01% packet loss. From [1]:
Unless an AQM node schedules application flows explicitly, the likelihood that the AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L). That is:
p_C ~= (p_L / k)2
The constant of proportionality (k) does not have to be standardized for interoperability, but a value of 2 is RECOMMENDED. The term 'likelihood' is used above to allow for marking and dropping to be either probabilistic or deterministic.
[0]: https://www.rfc-editor.org/rfc/rfc9330.html [1]: https://www.rfc-editor.org/rfc/rfc9331#name-the-strength-of-...
The actual limiting factor for how horribly bloated frontend code becomes is that at some point, it becomes so bad that it noticably impacts your business negatively, and you need to improve it.
Increasing the TCP window so it managed at least the basic asset delivery makes sense, but if you need to cold start regularly and you have hundreds of kb of javascript, perhaps fix your stuff?
She touched the TCP congestion algorithms of all TCP variants Tahoe, Reno, New Reno, Carson, Vegas, SACK, Westwood, Illinois, Hybla, Compound, HighSpeed, BIC, CUBIC, DCTCP, BBR, BCP, XCP, RCP.
And it all boils down to:
* how much propagation delay are there,
* how long are each packets, and
* whether there are sufficient storage space for “buffer bloats”.
Also, TCP congestion algorithms are neatly pegged as
* reactive (loss-based)
* proactive (delay-based)
* predictive (bandwidth estimation)
https://egbert.net/blog/articles/tcp-evolution.html
also citations:
DUAL (Wang & Crowcroft, 1992) https://www.cs.wustl.edu/~jain/cis788-95/ftp/tcpip_cong/
TCP Veno (Fu & Liew, 2003) https://www.ie.cuhk.edu.hk/wp-content/uploads/fileadmin//sta... https://citeseerx.ist.psu.edu/document?doi=003084a34929d8d2c...
TCP Nice (Venkataramani, Kokku, Dahlin, 2002) https://citeseerx.ist.psu.edu/document?doi=10.1.1.12.8742
TCP-LP (Low Priority TCP, Kuzmanovic & Knightly, 2003) https://www.cs.rice.edu/~ek7/papers/tcplp.pdf
Scalable TCP (Kelly, 2003) https://www.hep.ucl.ac.uk/~ytl/talks/scalable-tcp.pdf
H-TCP (Leith & Shorten, 2004) https://www.hamilton.ie/net/htcp/
FAST TCP (Jin, Wei, Low, 2004/2005) https://netlab.caltech.edu/publications/FAST-TCP.pdf
TCP Africa (King, Baraniuk, Riedi, 2005) https://www.cs.rice.edu/~ied/comp600/PROJECTS/Africa.pdf
TCP Libra (Marfia, Palazzi, Pau, Gerla, Sanadidi, Roccetti, 2007) https://www.cs.ucla.edu/NRL/hpi/tcp-libra/
YeAH-TCP (Yet Another High-speed TCP, Baiocchi, Castellani, Vacirca, 2007) https://dl.acm.org/doi/10.1145/1282380.1282391
TCP-Nice and other background CCAs https://en.wikipedia.org/wiki/TCP_congestion_control
TCP-FIT (Wang, 2016) https://www.sciencedirect.com/science/article/abs/pii/S10848...
I think cable modems have had a ton of improvement, and more fiber in our diet helps, but mobile can be tricky, and wifi is still highly variable (there's promising signs, but I don't know how many people update their access points)
(wifi also dampened bandwidth demand for a long time - it didn't make sense to pay for faster-than-wifi broadband)
Wifi is still mostly shitty in most places in the world.
Then there are countries like Philippines with just all around slow internet everywhere.
Or worse, different service tiers from a neighborhood getting bundled via CGNAT; though that's a clear argument for IPv6.
Sure, wireless is complex, but there were definitely some way too big buffers in the path. Add in some difficulty integrating their box into my network, and it wasn't for me.
There's a lot of issues and variables at play; this isn't a case of "it's always DNS". What tools do ISPs even have at their disposal and how accurate are they and does it uncover the actual problem users are experiencing? This is the real issue that ISPs of all size have to deal with.
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
In most western countries, the bufferbloat typically occurs at people’s home internet connections. As is the case in all cases of buffer bloat, the solution is to be willing to drop packets when the connection is saturated. If you limit the bandwidth just below what the connection can handle, you can do active queue management to solve the problem.
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply to continue your crusade. :/
It is not yet a "solved" problem, but 10-15 years have started to make a dent and get better tools to both observe and act on the problem.
This is seen everywhere from the inclusion of CAKE ( https://man7.org/linux/man-pages/man8/tc-cake.8.html ) in some CPE / home router, but the use of fq_codel ( https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html ) in routers along the way.
Other ISPs have to go even farther, because "content" might be 80-120ms away, and the ability to be more aggressive or less aggressive in tuning certain parameters can have a large impact on overall customer Quality of Experience. If there are any LEO hops along the way, problems with TCP and delayed signaling as a byproduct can also make throughout tank while latency spikes.
DPDK and VPP have contributed to a lot of new networking devices to help observe and act on traffic.
Everytime you go from a big pipe to a small pipe (higher data rate to lower data rate) connection you will see this issue at varying levels.
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
Out of curiosity, did you have full observability of these peering links, or is this a hypothesis? I could think of a few scenarios where alternative explanations could explain what you're seeing.
In most western countries, the bufferbloat typically occurs at people’s home internet connections.
Says who? How is this measured? Do we have actual numbers on people experiencing real bufferbloat issues that are affecting their service?
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply out of ignorance. :/
Look, I will cordially suggest a more tenable approach: consider disengaging from this thread, your vacuous and vapid post hasn't really brought anything to the table.
Edit: Seems I can't reply to the child comment, so I'll just say, you should've used your own advice and not reply. There's nothing of substance and you're still continuing with your daft misinterpretation of my take. I'll leave it at that.
The actual problem is I'm on a voip call and someone starts a big download (steam) and latency and jitter go to hell and the call is unusable. Bufferbloat test confirms that latency dramatically increases under load. Or same call but someone starts uploading something big.
If troublesome buffers are at the last mile connection and the ISP provides a modem/router, adding QoS limiting downloads and uploads to about 90% of the acheived physical connection will avoid the issue. The buffers are still too big, but they won't fill under normal conditions, so it's not a problem. You could still fill the buffers if there's a big flow that doesn't use effective congestion control, or a large enough number of flows so that the minimum send rate is still too much; or when the physical connection rate changes, but good enough. Many ISPs do this, and so you hear a lot less complaining about bufferbloat on say Comcast these days; also, this is an effective best practice, so less need for papers, reports and case studies... it's a matter of getting the practices in the wild and maybe figuring out how to do it better for wireless systems with rapidly changing rates.
Otherwise, ISP visibility can be limited. Not all equipment will report on buffer use, and even if it does, it may not report on a per port basis, and even then, the timing of measurement might miss things. What you're looking for is a 'standing buffer' where a port always has at least N packets waiting and the buffer does not drain for a meaningful amount of time. Ideally, you'd actually measure the buffer length in milliseconds, rather than packets, but that's asking a lot of the equipment.
There's a balance to be met as well. Smaller buffers mean packet drops, which is appropriate when dealing with standing buffers; but too small of buffers leads to problems if your flows are prone to 'micro bursts', lots of packets at once potentially on many flows, and then calm for a while. It's better to have room to buffer those.
Something I have always done I actually provision to account for packet overhead, so you might speed 2-3% higher speeds than your plan limit in a speed test, but psychologically the customer is getting more than they paid for, and most seem to be very happy about that.
But, rate limits were already in place long before anything about queue depth was even discussed, so that was nothing new. CAKE OTOH has had a very noticable impact on the customer experience, when their kids XBox can download that 250G update without impacting the voip call or wifi offloading another member of the household is on. Alternatively, that same gamer can play while Mom is downloading something near max throughput without having latency spikes and packet loss.
Yes, you're on to something about the customer experience in general that I'm tracking down myself. Orb is also trying to get a look, but I'm not a fan so far of that tool/platform https://orb.net/
By the way, when I wrote in another comment that bufferbloat was solved, I meant it in the same way that IPv4 exhaustion is solved by IPv6. We have the ability to deploy solutions that largely fix things, but whether we do is another matter. You are right to say that the past 10-15 years have started to make a dent in the problem. I had not meant to suggest otherwise.
I was being generous by advising you to stop posting, since the more you post asinine things, the worse you look. In the past, I have taken the liberty to do amateur psychoanalysis of people who post bizarre things online based on a psychology class I took in college. If I keep responding, it will only be to get you to post more so that I can work out what is wrong with you for my own curiosity. I am probably not the only one thinking this.
Look, let's call this what it is: gatekeeping. Furthermore you deflect and avoid answering a real question. I don't think you actually understood the crux of what I'm saying and instead resorted to ad hominems and gatekeeping, but seeing as it went over your head, I will pose the question: does bufferbloat have more than a marginal affect on the Internet experience of end users in real world conditions (not in a controlled experiment), furthermore does it affect a significant population, as of today in the 2020s as opposed to circa 2010? I'm saying no to both; a good way to gauge whether it is still relevant is to see publications in networking conferences and journals or even discussions by the *NOG, and really it's just not there. I know there's obsession over CoDel etc. and I used to follow the late Dave Taht's evangelising about the issue, but put simply the numbers don't add up - anyways a simpler solution would simply to prioritise ICMP and UDP flows over TCP. Anyways, this is not your imagined crusade against bufferbloat, it's just a pragmatic assessment. I'll leave it at that, rather than deflect and attack, consider applying some emotional intelligence.
Historically and likely still in the present day (but not in my household as we use AQM now), whenever one person in a household does a large download, internet latencies shoot up for everyone in the household, which is also bufferbloat. Having to wait hundreds of ms per round trip brings us back to the 56k dialup days and the performance impact on interactive traffic is horrific. It is enough to make VoIP unusable. As others have told you, there can be other issues at the same time, but bufferbloat makes the issues worse. I cannot speak for others on the extent to which they are afflicted by buffer bloat, but adopting AQM had a night and day difference in performance of the internet connection in my house, since I often do big data transfers that previously would slow down basic web browsing for everyone in my house, myself included.
As for your conjecture that extant problems are visible in recent journal publications, journals have a selection bias. The idea that a problem’s existence is indicated by the degree to which people are publishing papers on it in journals is fallacious since the papers need to not just provide something new, but also be interesting to those running the journal (i.e. make them think that the papers would elevate the status if their journal and help them get increased readership, provided that they are not a junk journal that will publish literally anything). On top of that, the work needs to be funded. Bufferbloat, which is largely considered a solved problem and which predominantly affects the less affluent these days, is not something that will get much attention in journals since nobody in academia seeks funding for something that they do not think they can improve or publish.
Finally, I did not use any ad hominem remarks toward you, as my remarks had focused entirely on what you wrote. I did write that any further replies would likely be done to get you to keep talking so I can play my old game of “figure out what is wrong with someone posting bizarre things on the internet”. About 30% of the population is mentally ill and thus when someone is posting bizarre things online, it is often the result of mental illness. Figuring out which mental illness is often the only reason responding to bizarre posts is worthwhile (as it is both an intellectual challenge and a public service). This contradicts your remarks suggesting that there is no point to my replies, to use my words rather than yours. It is not an ad hominem remark to say that I am likely to do this analysis. Posting the results of the analysis would be, but it would be grounded in fact and would likely be done to suggest professional help for X condition, if my amateur analysis identifies a condition that could benefit from professional help. Honestly, I think the world would be a better place if more people who studied psychology (even 1 class like I did) played armchair psychologist when others persist in a pattern of bizarre remarks and refer those who need professional help to trained professionals.
I think you suffer from tunnel vision here, particularly if you ascribe the issue to your ISP which would have magnitudes more capacity than subscriber links, even if oversubscribed. For Bufferbloat to be an issue in that regard they'd have to be a choke point, in which case there's actual seriously problems at that point. I'd expect being China there's a lot more going on anyway with GFW and poor routing.
Bufferbloat, which is largely considered a solved problem and which predominantly affects the less affluent these days, is not something that will get much attention in journals since nobody in academia seeks funding for something that they do not think they can improve or publish.
I feel this is the point you missed when having a conniption; I didn't say bufferbloat didn't exist, I said it was overhyped. I would like you to reflect on these two views and see how they differ significantly, as it clearly went over your head on your crusade to crucify me.
About 30% of the population is mentally ill and thus when someone is posting bizarre things online, it is often the result of mental illness. Figuring out which mental illness is often the only reason responding to bizarre posts is worthwhile
Sure happy to say I suffer mental illness. I have been morbid less than 0.5% of the time in the last 15 years, otherwise it is in remission. What's actually bizarre is thinking there is one type of mental illness that makes people somehow unhinged or can't carry a debate because you simply do not like what they say, it's clear that your one psychology class which you use as a crutch to spout nonsense shows you're clearly out of depth here, you should seriously stop; it's also abundantly clear you use it as a guise to gatekeep and to condescend; if you can't debate or argue in good faith, I would put forth the advice you keep trying to palm off - do not continue posting, disengage.
But if I didn't sway you with above, stop and at least read the HN guidelines; this isn't the other social media platforms you're used to.
I will point out that you were not interested in debating in good faith, as your sole purpose is to make the point that people should treat buffer bloat as a theoretical issue that does not actually affect people, with the explanation for every incident of buffer bloat always being something else. Everything you wrote has been consistent with that. You even applied a pattern of overanalysis to the discussion that can be used to claim the contrary to anything. Your apparent need to deny the existence of a well known issue is bizarre. You would be much happier right now if you had dropped the subject.
By the way, peering links between the greater China area and the rest of the world are notoriously bad due to limited capacity. China in particular has three tier 1 networks that all refuse to upgrade peering links between each other in a timely manner, and the peering links between them and the rest of the world are often just as bad as the peering links inside of China. In order to get a decent connection there in 2016, I had to forcibly do my own routing through VMs in data centers in Shanghai, Tokyo and others to control the links used, after much trial and error, yet there were still periods where the links were horrendous and surprisingly, the issue was not always exclusive to China according to my testing between VMs on various data centers. I was visiting family for a month, yet needed a decent connection to work remotely, so I spent an entire week non-stop studying the connectivity and experimenting with ways of improving my VPN connectivity. I also had proof that the GFW was not where problems occurred, since my tcpdump pcap files taken at VMs in Shanghai and Tokyo showed the packets traversing the GFW. I also had reproduced similar performance issues using VMs in other countries, such as Japan and Singapore, where there is no GFW, as part of attempts to identify paths over which I could forcibly route my traffic via iptables rules. It is obvious to me that you will continue to deny bufferbloat was a problem and instead blame something else, yet as others have already explained to you, bufferbloat makes problems worse. After a delay of a few hundred ms, my VoIP session did not care if the packets were delivered or not, since at that point, the packets were useless, yet enormous buffers would delay them and other traffic well past a sane expiry instead of dropping it, to the detriment of all trying to use the peering links.
It has nothing to do with the topic, doesn't add substantive discussion to the topic? The bizarre part is you keep bringing up mental illness without actually specifying one. You should just admit to yourself that you use it to skirt the rules, condescend and gatekeep. Please read the HN guidelines and keep that kind of rhetoric to other social platform mediums where that is condoned.
I should point out that you were not interested in debating in good faith, as your sole purpose is to make the point that people should treat buffer bloat as a theoretical issue that does not actually affect people
I've been debating in good faith, but when you repeatedly misinterpret what I'm saying and continue to argue for an imagined argument, I give up.
I should point out that you were not interested in debating in good faith, as your sole purpose is to make the point that people should treat buffer bloat as a theoretical issue that does not actually affect people, with the explanation for every incident of buffer bloat always being something else as if buffer bloat were not a factor. You were applying a pattern of overanalysis to the discussion that can be used to claim the contrary to anything. I really do not understand your apparent need to deny the existence of a well known issue, but it is bizarre in a bad way. You would be much happier right now if you had dropped the subject.
I'm actually fine; I just don't take well to pointless tangents, ad hominems, and curmudgeon behaviour which is really all your doing. It took you several posts to get to the content of what I was asking. As for overanalysis; is it though? A lot of people do not know how to analyse a problem in the first place, ignore variables, and jump to conclusions due to confirmation bias. You're taking this personally as you think I'm underestimating your ability to diagnose the problem, but that's not what I'm getting at and is central to my argument, there is a variety of problems on the Internet and it is hard to actually distill and get an accurate diagnosis. Experiencing latency or jitter is a tale old as time; and I think the overarching argument I've been pointed out is that we've improved Internet infrastructure for the most part that bufferbloat just isn't relevant (as it is hard to manifest and trigger), yet still gets hyped as an issue. You even conceded this point earlier.
It is obvious to me that you will deny bufferbloat was a problem and instead blame the peering links
I didn't deny it, I just said that you don't have enough information to argue its case, especially with the lofty argument that it is occurring at the ISP as opposed to your CPE. Let alone your ISP in China which immediately raises red flags with all the other issues that are at play.
Anyways, I'm kind of done, you're imagining a villain you must slay, resorted to gatekeeping, ad hominems, pointless tangents, etc. I'll leave you to it.
This is the most bizarre choice of “hill to die on” that I have ever seen.