Our goal was to improve compression by combining and tweaking the best aspects of LZ4 and Snappy.
The package provides Block (up to 8MB) and Stream Compression. Both compression and decompression have amd64 assembly that provides speeds of multiple GB - typical at memory throughput limits. But even the pure Go versions outperform the alternatives.
Full specification available.
Block and Stream Format: https://raw.githubusercontent.com/minio/minlz/refs/heads/mai...
But comparisons with zstd & lz4 would be nice.
Is someone planning on a wuffs/rust implementation with C API so that it wouldn't be Go only?
I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C implementation.
Having been 10+ years since I've done any serious C, I didn't want to assume I could write a good C API for it.
If anyone are interested in picking it up, reach out.
> comparisons with zstd & lz4
The repo has lz4 comparisons (Go version) for both blocks and streams.
I see MinLZ more like a compliment to zstd, just like lz4 - where speed matter more than ultimate size.
Here is the fully parallel speed of block decompressing with zstd or minlz - Go versions:
* Protobuf - zstd: 31,597 MB/s - MinLZ 155,804 MB/s. * HTML - zstd: 25,157 MB/s - MinLZ 82,292 MB/s. * URL List - zstd: 16,869 MB/s - MinLZ 45,521 MB/s. * GEO data - zstd: 11,837 MB/s - MinLZ 36,566 MB/s.
Of course zstd compresses to a smaller size - but for things like streams transfers or for fast readbacks from disk you probably want the fastest.
Overall, I would say that MinLZ is about 2x faster decompressing, and compresses about 2x the speed compressing at max speed. But of course with less compression.