Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache(blog.cloudflare.com) |
Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache(blog.cloudflare.com) |
leaves
What's the speed of service/response time relative to the data source?
At that point it might be enough to replace your multiple caches with fewer in-RAM databases?
It's an interesting problem.
Not really, TTLs are often short, but IPs might not change for years.
You can probably generate your own TTL, at scale, and avoid many DNS requests.
The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.
[1] Nobody has to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
This is an incorrect statement. Caches do not have a requirement of being smaller than their source data set. CDN is an example of a cache that generally matches the size of the source data.
Were there no design discussions/reviews when the system was setup to catch trivial things like this?
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.
Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.
Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)
Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.
Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.
https://web.archive.org/web/20260314210910/https://users.ece...
Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.
It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
So you agree that they should've designed the system to use the appropriate data structure from the beginning?
At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
So how would you decide which path to take in situations like this?
Sometimes it's a lot of small things everywhere and you can pick up significant performance after a lot of small value fixes. In this case, caching wire data instead of structured data is almost one of these, because the contribution to response time for serving a cache hit is small... otoh it happens so often than a small improvement matters; but this is a pretty focused use case, you usually hit the many smalln improvement issue in a less focused application where there are many code paths.
Sometimes the whole code structure / data structures are so wrong, but it works and perf is bad and profiling will never tell you. This article is not that case; these data structures only needed refinement.
if you spend cycles on nitty gritty opinions like this time to market goes out further and further out. some napkin math, 130 gen13 servers cost "only" ~$2.6M. relative to the importance of the 1.1.1.1 and the market at the time. that is nothing to cloudflare.
this is not to say good system design does not matter. it very much does, but making that call at that time would've butchered the prodcut very much similar to google+, youtube etc.
It's also not nothing, otherwise it would never be optimized away now, but left as is. After all, wasting time on optimization delays "time to market" for other useful features.
I also don't get the reference to YouTube, it's a very successful product, how was it butchered by good system design???
You're tasked with making a DNS caching recursive resolver that can operate at a large scale and will be run on thousands of servers each of which has a lot of GBs of ram.
You are given some period of time to build this and make it production ready. How do you spend your time:
* Focusing on making sure that the resolver works correctly?
* Focusing on make sure that it actually provides improved DNS performance for internet users?
* Handles an very large number of record requests/s?
* Saves a few GB of ram per server?
There are tradeoffs to consider. RAM is cheap, even at today's prices RAM is not the most expensive thing that can go wrong in such a scenario. Having the responses be slow or incorrect is a far more expensive problem. A good engineer would pick a simple data structure that has the right shape but might not be optimal in footprint to focus on correctness and response time. The few extra GBs of RAM per server can be dealt with later.
When building things at scale you want to make sure it works correctly, fails correctly, and does the thing quickly before worrying about reducing resource consumption. I've never seen a project fail on Vec<T> vs Box<[T]> memory differeneces, or even on a few GBs of RAM usage per instance. I have seen them fail on "one wierd corner case of correctness" though, and on poorly thought through failure modes.
Another interesting thing that happens is you don't necessarily know what form your actual optimizations will need to take. Later when your systems grow you discover the suboptimal parts you hadn't optimized for.
Very early on at Cloudflare I worked on part of the DNS infrastructure that took DNS records from the UI and got them in a state for actual authoritative serving. The system had been constructed anticipating Cloudflare having millions of customers with unique domains, but it had not been constructed for a single customer with a single domain with millions of records. This caused a periodic slow down in DNS record updating while the system churned on that one customer.
In a different job I worked on a piece of optimization software that needed to keep track of "node" A is reachable from node "B". This had been implemented as a matrix (literally a malloced NxN matrix of ints storing 0 or 1) which worked really well for small systems. But you'd be out of memory really fast on a large project. I replaced the matrix with a hash table and all was good because the matrix was actually really sparse.
With a rather short prompt, claude/codex will take your code, write a harness, profile it, build experiments, profile those, and give some pretty solid advice which one to pick. Then integrate the changes. It's the kind of goal-directed, bite-sized job that LLMs excel at. Extremely low-commitment.
Except for the whole "making changes in production at scale" problem, of course.
The “evil” of premature optimization is that it’s a misapplication of priority. If I have an acute medical problem that needs attention, it’s not the right time to talk about chloresterol and statins, get my broken leg set.
There’s always a tension between engineering management who needs to deliver a solution to the business and engineers who want to deliver a beautiful object.
Because anyone willing to come in just to design your cache format is going to expect payment that is many multiples more than the engineers you already cannot afford? Long-term employees cost less, which brings them closer to being affordable, but you have to be able to keep them busy for long periods of time to realize that reduction in cost. A engineer who doesn't understand your codebase isn't going to be useful for very long.
Doesn't this also inform you that your cache will be very large, so you shouldn't use growable structures with slack space when cache entries won't grow; slop space reduces the size of your cache. And also that the query volume will be high so the cached data should require as little work as possible before returning data; spending time marshalling response data on every cache hit increases response time and decreases capacity.
* unbounded growth of the cache and properly invalidating after TTL expires (a few GBs of slop is nothing on a server with 64 or more GBs of ram, unbounded growth is a problem).
* making sure the DNS implementation works correctly on both the serving side and recursive resolution side.
* What strategy is best for deduping recursive requests across machines (if something a few miliseconds away has a live result, why do a full lookup taking hundreds or thousands of milliseconds?). This potentially improves RAM usage across the datacenter too from not having a given record on dozens (or more) machines' local cache. I don't know exactly how they do it, but naively I'd look at some sort of DHT shaped solution to look for records in peers within the datacenter. Or maybe some sort of tiered caching with the upper tier being sharded on domain name or the like.
* The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit. This is on the order of 10s or 100s of ms due to having a big cache and short distance to the requesting machine. A suboptimal lookup algorithm that is a few microseconds slower in local compute and ram access is just not as important as the other concerns for dedup and cache sharing. That's not to say it's unimportant, just that it's not the top priority when you're trying to deliver this much larger performance gains from other aspects of the system. Thats why they are getting to it several years after release.
Cloudflare writes a lot about distributed systems solutions to various problems. They likely don't think as hard about single machine performance as much as whole datacenter performance when approaching problems.
Keep in mind that the per-server cost of the whole program pre-optimization seems to be about 10GB (from the graph in the post). IME that's not bad for a big busy caching service.
Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped).
When you're storing billions of records, it makes sense to spend a few minutes to consider how they're used and make a good choice about how to store them.
When you're getting a cache hit tons of times per second, it makes sense to consider every step and which ones don't need to happen every time. You have to consider every step while you're pursing correctness anyway, so might as well have the performance lens active too.
I'm not asking for heroic optimization: I didn't ask for vectorized stuff or kernel/nic offloading or kernel bypass networking... Just you have to use some data structures, you might as well not use ones that are expensive for features you don't need; and you have to store something in your cache, you may as well store something that requires less munging on the way out.
If this were a small local cache, that didn't want to use something already existing like unbound for some reason then yeah, data structures don't make a huge difference, extra marshalling doesn't make a huge difference, just don't reimplement all the CVEs that BIND had in the 90s. But if you're going to allocate 100 TB of ram, make it count. Even if you do use twice the ram but you get value from it, maybe that's fine... I've run wacky systems with bloated storage when there was a benefit. Vec doesn't give any value over a Box<[]> in this case; convenience or lazyness would be fine except that the sheer number of objects makes it worth the few minutes it takes to do something better.
This is true. I'm arguing that its unlikely this was ever bound by available RAM. Cloudflare is a DDoS protection company that absorbs attacks. They have a lot of available capacity at any moment. When you're building a service in a sitaution where you have more capacity than you'll likely need.
The savings were 100 TB across >300 data centers. The savings were on the order of 50%. So prior to this reduction the service was using something less than 2/3 of TB per datacenter. The service ram usage was about 10GB per instance according to the graph in post. IDK how cloudflare divides thier stuff between machines, but assuming they don't run less than 64 GB per server that's less than 12 servers per datacenter of ram for a flagship product, and they likely run it spread across 65 of the machines in the datacenter that are also doing other stuff. The per-instance RAM likely isn't the the concerning limit.
Overall RAM usage is proabably a bigger concern. Thats why I would think about dedup between instances and distributed caching strategy first. I could focus on redudcing the ram needed per service instance and get a 50% reduction per machine. Or I could focus on deduping 1/n (where n > 2) reduction in total memory usage across all instances. Personally if I was worried about reducing RAM I'd put more energy into growing N.
However all this is a red herring. The assumption people are making is that the cache was always read-only, and it's obvious that Box<[T]> was the best decision because in a RO cache smaller entries hold more things.
The 1.1.1.1 service advertises improved DNS performace. That's its value add. The biggest performance gain you can have from a cache is not having a cache miss, and in DNS a cache miss means a very expensive recursive lookup. So there's concerns about how to minimize those lookups. If one instance has does a lookup, it makes sense to share that result to the other instances that may need to do a lookup [1]. I don't know off the top of my head if it makes sense to get those updates and modify the existing record or just replace it in the local cache. That comes down to locking strategies and reading patterns in the specific code and service traffic patterns. Until i have hard evidence one way or another I'd like my cache to be able to do both and keep the data structs modifiable until that's nailed down. If per-isntance ram ever becomes the issue, there's easy wins there to buy me time to find better large scale solutions to the problem.
No one is disagreeing that the larger datastructures are larger. No one is disagreeing that they take more RAM, and or even if RAM was the the problem reducing it would be good.
The thing people are pointing out is that this isn't a homework problem about an optimal cache structure in a vacuum. We're pointing our that engineering real large scale solutions has a lot more to consider than a homework problem, and that the thing you're harping about likely didn't have any real budgetary or noticable performance impact on bulding that system. The reduction in ram is just a smallish improvement in operating costs after all the more expensive stuff was figured out.
Put another way 100TB of RAM is ~$350K. Thats one engineer year for a mid-level engineer.[2] Would you rather spend that money to save an equivalent amount of money somewhere, or... would you spend that money putting the engineer on something that saved $700K elsewhere (alternately that generated $700K)?
[1] I talked a lot about dedup and the simple gotcha is "hahah then its not deduped so you need smaller objects". But on a service that is running on a few dozen instances having a few redundant copies to deal with loss of a machine and/or load can still result in 1/(n>2) savings in total ram.
[2] I'm not saying someone worked on this for a year btw, a couple people likely spent a couple months on the code, validation and testing of it. A manager spent time overseeing it. Operations people spent time understaning any effects it had on running systems. Costs add up and it wouldn't suprise me if this didn't end up being roughly break-even for the year.
Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
You can define away ‘stale’ by picking a consistency model, but look inside the consistency machinery and you will see fresher data you aren’t allowed to have yet.
https://www.rfc-editor.org/info/rfc8767/
As a corollary, there is obviously no floor on refetching unexpired RRs, of course, except for efficiency concerns.
At least in my country (UK) I know of no law relating to DNS caching.
Why throwaway perfectly good data every few minutes that is only modified every couple of years, just so someone can move their domain quickly when they eventually wish to? It is my contention that a [caching] DNS service can do far better. Trusting user (domain owner) input blindly is not for me.
I should be a bit clearer here; the TTL is an upper bound on how long it can be cached. Caches are free to consult more frequently but not less frequently. That said, out of respect for upstream cache operators and authoritative servers, most DNS caches honor TTLs as best they can.
You asked ‘why would I want to use a service that serves stale data’. The answer is you don’t want to. But if you have a distributed system your only choices are about the nature of the possible staleness, since global instantaneous consistency is known to be impossible due to fundamental features of the universe.
My original question could also be described as asking why I would choose to shop at a grocer where items on the shelf are routinely far past the printed expiry date. That's perhaps a more direct analogy.
Serving up records with TTLs in the past is so deeply weird that I didn't even consider that was what the grandparent had in mind.
Advocating to do things against agreed-upon standards without a compelling reason and without giving due consideration to the adverse consequences is one of the hallmarks of a bad engineer. Even Microsoft played nice with Internet standards for the most part (although with some notable exceptions at the application layer that got them well-deserved criticism).
The grocery store down the road only sells me things for profit but I hope you're not trying to imply that's somehow nefarious.
Besides, not all profitable ventures are bad.
Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.
In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia
This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.
However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.
So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
Pretty sure they were joking.
<strange high pitched equivocal sounds>
I do a lot of tuning work that others either can't be arsed to do, or more often are scared shitless of touching (learned helplessness). I tend to work on things that have to go right the first time in general, but I firmly believe that I learned that skill doing optimization work early in my career, not the other way around. Through a series of unfortunate events, my first specialization was performance, before I got good enough at anything else for people to listen to me.
Ultimately, the single thing that fucks up performance work, far and above any other concerns or sources, is that the vast majority of developers want to skip from Make it Work to Make it Fast without going through Make it Right. Then they have to use dodgy 'optimization' tricks that embrittle the entire thing, destabilize production and make every new feature feel like walking a minefield.
So what happens is that when someone who actually knows what the fuck they are doing comes along and wants to build a superhighway, they find the route littered with abandoned 'vehicles' that have sunk into the mud up to their door handles and left to cement in place. Clearing the proverbial route becomes a bigger technical challenge than building the road. And so you'll find that your peers have a bunch of ideas about what needs to be done but none of the stomach to do any of it.
Which is why I say "stubbornness" is the most important tool in my performance toolbox, far above cunning.
I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.
Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.
I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
We give perverse incentives to the marketing department to bring us the most customers they can, instead of the most appropriate customers.
Someone told me at my second lead position that they had landed a top-tier customer for our demo-ware and the first words out of my mouth were, "FUCK ME". Not the response they were expecting, but then that guy never did end up understanding me the entire time we worked together. My bosses did though.
When your system is new you're a loss leader for all of your customers. Every dollar they bring in costs you two, possibly more. You have to get to positive MRR before the venture capital runs out, and over a long enough time horizon, you will go for a new round and find out that a recession is about to start and the VC guys getting cagey is the first clue it's coming.
Meanwhile if you spend all of your time and energy on reducing costs instead of increasing revenue, then your competitors catch up with you. Particularly if their funding rounds are half a cycle off from yours - one of you will be the last one to have gotten a cash infusion before the money dried up.
Most recently I worked at a place that didn't understand my calls for sobriety until it was too late. So I got put in charge of a rear-guard action that was too little too late, and our customers all fled to much cheaper competitors who could do 80% of what we could for half the price. I learned some good stuff, but the company got bought by a competitor who scrapped those systems.
this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.
Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.
Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.
Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.
not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
It's only right if you are not the one who pay the bills.
from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.
its not wrong in my eyes, but definitely not the only path to take.
Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...
I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.
Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.
https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...
My personal issue is the misogynists who have created a hateful completely false narrative that 80% of the women sleep with 20% of the men (including the very demeaning and hurtful notion that all women are sexually promiscuous, but only if you’re one of the 20% of supposedly “Alpha” men) [1] Twitter is also full of—let’s call a spade a spade—racists who constantly post some video from years before showing some random Black person doing a criminal act, and then a bunch of racists comment that that’s how all Black people are and it’s the “evil left wing media” suppressing this supposed “truth”.
Just as Twitter has become a right-wing cesspool, Reddit has become a leftist cesspool, so I also avoid Reddit, which, like Twitter, is also becoming a closed walled garden—they just this month started clamping down on people reading old.reddit.com anonymously, so now you have to log in to have a usable interface with Reddit. Excuse me, no.
[1] This annoyed me to the point I researched the claims to verify it’s a bunch of bullshit. https://samboy.github.io/blog/80-20-myth.html
- TigerBeetle: A database without dynamic memory allocation, https://news.ycombinator.com/item?id=33192288 (2022).
- Succinct Data Structures: Cramming 80,000 words into a Javascript file, https://news.ycombinator.com/item?id=2348619 (2011).
If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.
It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
Thank being said in this case it should be impossible to index out of bounds so maybe a panic is warented.
Not really. You just need to make the underlying fields private and provide methods to get slices to the data you need.
type Wasteful struct {
a int16
b int
c byte
}
type Aligned struct {
b int
a int16
c byte
}
In langages that don’t there’s a tension between memory use and human readability / consistency of the layout.
There are also other domains which can be affected e.g. databases, it’s a concern / issue when using postgres for instance as it uses aligned columns and stores them in schema order.
Interestingly, there is also no way to write a loop on i for all the values between 0 and 99 without specifying the order. Luckily, in this case, the compiler is allowed to prove that the order has no impact (because it's local), and to decide that it will scan the values in a different order for optimisation purposes.
So the compiler could do it on a structure as well, as soon as it's able to prove that the structure is not exposed in any way to any code that it doesn't control, but that's much more difficult than proving that variable i is not visible outside of a tight loop.
So most languages opt to follow what their predecessors did: do nothing and task developers with reordering the structure if they want to minimise its size.
The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
We had to implement a streaming system and figure out a good way of chunking the world. This was a challenge because everything was on water and you could see nearby islands quite far away. An intern called Tommi did a great job identifying a good strategy and writing the system.
We also had to reduce the density and model complexity of a lot of environmental details such as rocks, vegetation, and stuff like pots and clotheslines. This was done largely by sorting things by memory size and frequency of use and identifying outliers.
One of the biggest issues was actually really silly: the journal fetched Portrait images and names of characters by referencing the actual NPC and having them embedded there. Meaning the journal, which was always loaded, would pull every NPC involved in a quest into memory including their behaviors, textures, and models.
On my end I also found a lot of silly details wasting hundreds of megabytes. Special render passes using huge textures and render targets, poor structuring of the render pipeline caused memory increases, several key shaders referenced huge textures which weren't necessary, ...
My blog posts about the project[0] mainly focus on rendering performance because that's where I spent more time and it contains more interesting content for discussion. But reducing memory was an ongoing concern with countless little improvements over the 1.5 year porting process
Even a BTree with compressed prefix keys can save space in the qname.
Relevant support page, though light in details: https://support.mozilla.org/en-US/kb/captive-portal
Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.
Edit 2: Wikipedia says both methods are used: https://en.wikipedia.org/wiki/Captive_portal and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.
My point was: that domain is not treated any differently from other domains.
And they say nobody uses IPV6.
Alternatively a btree somehow they can take advantage of prefix compression
> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.
jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
There are many reasons a specialized allocator can be better, but just saying "write your own" doesn't really add much value to the discussion.
It is almost like nobody even thought during the design phase about what might happen down the road.
This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
In the case of internet infrastructure I don't think there was anybody even up to the year 2000 who had any idea of how bit this was going to be. And even now we have IPV4 and lots of legacy to deal with. Cloudflare is not my favorite company, let's put it like that, but in this case they show how the sausage is made and I think that should be applauded. Much better than 'why were down again for X hours'.
Assume a sorting algorithm as a metaphor for your whole program.
You make it work by implementing the simplest thing you know how to write: bubble sort. Works.
At the end you notice it's way too slow and replace with something much better: quicksort.
Now, how much of your program is surviving? Almost nothing, perhaps except for the "greater than" comparison.
If you apply that idea to a real world program, we're pretty much talking about a full rewrite.
every dept knows what they could do with more budget, the budget for those things just never comes
now agents have utilized budget more effeftively, unbottlenecking many things, including engineering blogs
Interestingly this is exactly how netlink works-ish: https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...
You start, get the type & length, and then that is how many bytes you read.
Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.
In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.
So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.
As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
Hey dang can I get my rate limit turned off pretty please?
[1] https://doc.rust-lang.org/reference/dynamically-sized-types....
I assumed they couldn't do that because they're using it with some kind of generic HashMap<K, V>. In that situation, can "V" be dynamically sized?
A dynamically sized "V" would mean you can't have an array of them, which might preclude some hash map implementations.
, which HashMap does not do, i.e. the keys and values have to have a statically known size.
Cloudflare started to pick Zig recently, for projects, that have memory constraints.
You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".
> Rust is not a good choice for this kind of tricks.
Rust can do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
The CloudFlare manually implemented a clumsy version of this.
Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?
I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
The right way is that there's DHCP option for the network to signal "I have a captive portal", that's been standardized for over a decade.
… or … IDK … just stop shoving ads down people's throats just because they want WiFi.
In a row oriented database, you get a contiguous spot for the whole row even when there are multiple variable width fields.
So logically you'd have the equivalent of:
struct FooRecord {
int fixed_sized_field;
char some_other_field;
string first;
string last;
string title;
}
struct FooRecord {
long __length__;
int fixed_sized_field;
char some_other_field;
char* first;
char* last;
char* title;
}
There are variants of the above, of course. Instead of pointers the compiler could use lengths, offsets, or a pointer to the end of the variable length field -- this works because the beginning of the first field is at a fixed offset, and then pairs of pointers delimit the rest.
You can rely on the heap allocator to track the "__length__" instead, or you can encode it into the record explicitly to make "dynamic sized copies" simple.
Windows APIs generally work this way! You create a buffer, put a length in the first field, and then the API call writes a fixed-sized prefix followed by the dynamic-sized fields into the buffer. The 'length' is replaced too, so you know how many bytes to copy out without having to understand the structure.
Database engines go one step further and pack multiple "records" into a single "row". They typically store the fields "packed" at the start of the row with 16-bit length or offset markers at the end for the various dynamic sizes.
Something like:
fixed_sized_field // Row #0
some_other_field
first
last
title
fixed_sized_field // Row #1
some_other_field
first
last
title
... empty space ...
next_offset // always populated
row#1_title_offset
row#1_last_offset
row#1_first_offset
row#1_offset
row#0_title_offset
row#0_last_offset
row#0_first_offset
row#0_offset // typically the constant zero
https://www.rfc-editor.org/info/rfc8910/ is, I think, the latest version of the standard.
Better performance = saving money + better user experience.
The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.
In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
Insufficiently optimized code kills companies in highly competitive markets.
I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.
I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.
> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
I still look after a few VMware estates and a lot of Proxmox ones (that used to run VMware).
Hilariously, VMware is described as "enterprise class", which I can only conclude means MVP and a bit wanky.
Today I repaired a Proxmox HA + Ceph node using boring old normal Linux skills and as it turns out I have 30 years of those. Part way through a remote v8 to 9 upgrade I think I lost comms due to using OpenvSwitch for networking and despite using tmux for the upgrade session. Anyway, the Proxmox ISO was useless for rescue but the classic systemrescuecd worked nicely and I could run dpkg in a chroot.
VMware "used" Linux and never really gave back. I don't miss fixing vCentres and all the other nonsense that "Enterprise" wankery has foisted on me over the years.
> "The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal"
also:
> "In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable"
In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.
In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".
Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you can prove cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
Tons in the $200s btw, and this is in the city less than 20 minutes from the Nelson-Atkins Museum, WW I museum, the river, farmers market, KC Chiefs / Royals / Sporting KC / KC Current teams, parks, schools, newly rebuilt international airport, etc.
[0] https://www.realtor.com/realestateandhomes-detail/3711-N-67t...
[1] https://www.realtor.com/realestateandhomes-detail/20118-E-Tr...
Acknowledging this isn’t always easy or possible, but just pointing out that this is a self reinforcing problem.
I meant desirable for me to live there, not as an investment. Who wants to buy a home in place they don't want to live?
I mentioned acres of land. You normally don't have multiple acres of land in the suburbs.
> is it possible to buy a reasonably nice home located in a reasonably nice amerikkkan city… for $300k in 2026?
Who wants acreage? We want homes.
Then I actually met some enterprise software, and realised that it means 'expensive', 'bespoke', 'one-off', and usually 'janky'.
Why solve the problem directly when you can abstract everything away into FactoryFactoryImplementationInterfaceFactorys, and have something that is both a memory-hog and completely unassailable to any normal programmer seeking to understand it or make changes?
That said, look around:
https://www.zillow.com/kansas-city-mo/?category=SEMANTIC&sea...
Between $175k and $350k center on KCMO alone there are over 760 results.
Note: your first link https://www.realtor.com/realestateandhomes-detail/3711-N-67t... is 23 minutes from the WW I museum.
Note 2: Your second link is way out in Independence, MO but is still only 29 minutes away from the WW I museum.
First if you us the convention center, you place the housing center off of where most of the housing actually is.
Between $175k and $350k center on KCMO alone there are over 760 results.
I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.
This is my direct quote.
Let me break this down:
1. That would buy you thousands of square feet
True, clearly seen in the search results.
2. and often several acres
True, you just filtered down to several results currently on the market.
> I realize multiple acres of land is a bit of a silly requirement
Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.
But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.
What does "city" mean to you? For some, it's 500 people, or 5,000. For some, it's 5 million. Define that first. The US is a big place, and I know people that don't live within 50 miles of another human.
Otherwise:
https://www.zillow.com/homedetails/424-Olive-St-Kansas-City-...
4 bed, 3 bath, 1,580 sq ft, beautiful! $342,500, built in 1900.
https://www.zillow.com/homedetails/3508-N-College-Ave-Kansas...
4 bed, 4 bath, 2,295 sq ft, $365,000, built in 2022.
Yeah that sounds cruel and messed up, because it is. I don’t like how expensive real estate is and I wish we could have another 2008 without a bailout. But the point stands. If you hafta pick between eating and paying your mortgage, you made a bad financial decision. Blame whoever you want, I am of the opinion that people still have agency and are responsible for their decisions, as unpopular as that seems to be these days.
You'd be surprised how much house and land you can get for that price.