We could save petabytes of cache storage with Zstandard and Pingora(blog.cloudflare.com) |
We could save petabytes of cache storage with Zstandard and Pingora(blog.cloudflare.com) |
For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.
There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
I just repeated the process. So directory of jars is 368M. If I just 7z it, it'll be 282M. But if I unpack them (1.9G), and then 7z them, it'll be 96M. Pretty substantial win.
The drawback is that you probably can't easily restore previous jar file byte-for-byte which might matter for some use-cases. I guess it's possible to achieve byte-for-byte copy with more effort.
It's definitely possible to optimize the rlib format, like representing generic types as a tree of short IDs instead of a string or separating them into an optional debug file. Or do it like Zig with a new IR designed to be compact and easy to assemble into final executables. Even a simple global cache for the times you get lucky and end up with the same hash could help.
The problem is no one on the core team has time to do it because they are always overworked from the amount of bugs that need fixing. They also have a culture of making sure all changes are perfect before they go stable as a overreaction to C++ shipping half baked proposals. I think these are the reasons everything is moving so slow.
It had the nice effect that whoever caused the headers to need to be recompiled was the only one who had to pay the recompilation tax instead of everyone.
You may be cleaving the problem along the wrong plane if you're worried about sharing between multiple projects on the same machine versus across all of the members of a single team.
Artifactory is older than dirt. And its main feature isn't even saving bandwidth, it's ensuring that versioned libraries can never be overwritten in place with a different version potentially containing a trojan. And it can be set up not to download new versions until they've survived long enough for someone to hopefully notice a supply chain attack.
It was part of the (now defunct) frea project: https://freajs.github.io/frea-website/
https://joshleeb.com/posts/chunking.html
https://www.usenix.org/conference/atc16/technical-sessions/p...
On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
> Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already.
It's because they're already lossily compressed that, precisely, they're the typical targets for transcoding.
EDIT: oops, wrote lossless instead of lossy, sorry about that
A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.
My reading - and I care because I’ve been using concatenation tricks in parquet - is that this is required by the standard.
In the distant past I made a jinja2-like template system for server side rendering that precompressed the static bits and injected the short dynamic bits into the output stream as literals with some basic lz match rewriting in the following chunks etc. This kind of trick ought be better with zstd frames.
They also don't always support window sizes above 8MB which means levels above 19 can fail to decompress (if they use the more than 8MB of window).
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
ie btrfs
You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted
It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
With a dictionary, small objects - even just a couple of bytes - compress well.
Compress the headers too, and suddenly it's worth doing
Zstd has a seekable format for frames, similar to pigz --independent works.
[1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
Turns out we can squeeze quite a bit more compression performance out of DEFLATE - ~10x in certain instances, 2x as a base minimum (read the paper for details).
Seekable OCI (SOCI) uses an index so I imagine that's an option (real byte range a-b maps to compressed range x-y). Presumably you'd still need to read the header and some additional pieces
Given how fast zstd can decompress, this may or may not actually be a win: the time spent waiting for I/O might be so large that the decompression can fit within the wait time.
You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again.
And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits.
So compression makes the most sense for something you want to keep a long time that will be accessed very rarely. And the least sense for something you will drop very soon and will have many people requesting it.
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source).
Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size)
"Ignore cold assets" makes more sense with that framing
Although if that's the case, the CPU statement still is a bit confusing.
This sounds like a security nightmare - there is a reason we have build servers (well there are many reasons, but build safety is one of them).
But I have to admit the idea is clever.
There's this, but it doesn't seem to be getting much traction: https://github.com/facebook/zstd/tree/dev/contrib/seekable_f...