S3 Express Is All You Need(warpstream.com) |
S3 Express Is All You Need(warpstream.com) |
Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtrips. S3 Express changes that equation dramatically, as S3 Express approaches HDD random read speeds (single-digit ms), so we can build production systems that don't need an SSD cache—just the zero-copy, deserialized in-memory cache.
Many systems will probably continue to have an SSD cache (~100 us random reads), but now MVPs can be built without it, and cold query latency goes down dramatically. That's a big deal
We're currently building a vector database on top of object storage, so this is extremely timely for us... I hope GCS ships this ASAP. [1]
[0]: https://github.com/sirupsen/napkin-math [1]: https://turbopuffer.com/
NVMe is what is changing the equation, not SSD. NVMe disks now have up to 8 GB/s, although the crap in the cloud providers barely goes to 2 GB/s - and only for expensive instances. So, instead of 40X better throughput than S3, we can get like 10X. Right now, these workloads are much better on-premises on the cheapest m.2 NVMe disks ($200 for 4TB with 4 GB/s read/write) backed by a S3 object store like Scality.
The comment you reply to is talking mostly about latency - reporting that S3 object get latencies (time to open the object and return its head) in the single-digits ms, where S3 was 50ms before.
BTW EBS can do 4GB/sec per volume. But you will pay for it.
This instantly makes a number of applications able to run directly on S3, sans any caching system.
Take it a step further, gdal supports s3 raster data sources out of the box for a while now. Any gdal powered system may be able to operate on s3 files as if they are local.
This is key insight. The data storage cost essentially becomes negligible and latency goes down by a magnitude by making S3 Express as a buffer storage then moving data to standard S3. I see a future where most data-intensive apps would use S3 as main storage layer.
While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter.
I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they have a lot of requests.
Unfortunately the price model puts it in a place where it is the right technology only for some very rare places.
In a nutshell the key thing you need to know is: - The storage is 6.4x expensive than classic S3. - The GET requests are 2x cheaper (with additional cost for large requests). - Your data is replicated within a single region. - latency is single digit ms.
From a pure cost wise point of view, the realm where it makes sense to use it is there, but small, and often competes more with EBS than it competes with S3.
IMO warpstream is a really cool product and this new S3 offering makes them even better
> Engineering is about trade-offs, and we’ve made a significant one with WarpStream: latency. The current implementation has a P99 of ~400ms for Produce requests because we never acknowledge data until it has been durably persisted in S3 and committed to our cloud control plane. In addition, our current P99 latency of data end-to-end from producer-to-consumer is around 1s
via https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk...
However, we now install by default to local disk filesystem, since it's much faster to just do a periodic S3 hot sync, like with restic or aws-cli, than to treat S3 as the primary backing store, or just version the EBS or instance volume. The other reason you might want to use S3 as a primary is if you use a lot of disk, but our files are compressed and extremely small, even for a large installation with tens of thousands of users and instances.
0. https://userify.com (ssh key management + sudo for teams)
I ask because I've always been taught to not store files in a database. This use case sounds like an interesting exception.
There were some benchmarks, I couldn’t fine where SQLite was faster than native file system at retrieving, searching and adding files to a large directory.
The new storage class is SSD-backed, presumably doesn't use Java anywhere, and doesn't stripe your data across as many hosts. It's more expensive because SSDs are more expensive than HDDs and narrow erasure codes are more costly than wide erasure codes.
Source: Used to work on S3.
Or is rust really that magic?
https://azure.microsoft.com/en-us/blog/premium-block-blob-st...
And a second question, would it be worth the 8x times surcharge?
> S3 Express One Zone can improve data access speeds by 10x and reduce request costs by 50% compared to S3 Standard and scales to process millions of requests per minute.
I get it, but at the same time that is also what you lost when you locked yourself in with a particular vendor.
What are other viable practical alternative solution(s)?
Off the top of my head, Backblaze B2, Cloudflare R2, etc are S3 compatible, and Minio locally.
If you have a big use case and you really understand your needs, it's very doable.
Before doing that it was unacceptably slow. After doing that it was unacceptably expensive.
treating storage as an application-controlled thing that doesn't need systems management is a good thing. i want "put this file in this spot" logic in my application code, not "put this file in this spot on the filesystem, and hope that location is backed by the correct storage layer"
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountp...
For example a container or EC2 instance might only need a tiny bit of your storage and with s3 can just download what it needs when it needs it.
As opposed to EFS where the container or instance needs to load in the entire datastore on startup which can add minutes to startup time if the EFS drive is large.
We also saw about a 10x speedup by moving all caching into the server process, and since it was all in the same process, we no longer had to compress and encrypt data before sending to Redis. We still checkpoint the moving server state, encrypted and compressed, to disk every sixty seconds, just like Redis would do with BGSAVE, so we can start back up within a few seconds (actually faster than the old Redis after a restart.)
Furthermore, a single SQLite database holding 10-kilobyte blobs uses about 20% less disk space than storing the blobs in individual files.
The performance difference arises (we believe) because when working from an SQLite database, the open() and close() system calls are invoked only once, whereas open() and close() are invoked once for each blob when using blobs stored in individual files. It appears that the overhead of calling open() and close() is greater than the overhead of using the database. The size reduction arises from the fact that individual files are padded out to the next multiple of the filesystem block size, whereas the blobs are packed more tightly into an SQLite database.
[1] Yes, I am aware of multi-attach but this introduces a scaling bottleneck and requires a fairly exotic setup.
2. Variable throughput may be a concern.
3. You may have availability concerns.
It was an ok solution for what we were doing, but several times I came really close to just dumping it and standing up an NFS server using EBS volumes.
I also used it a couple of times to store webroots and that was a complete disaster with systems that had lots of small files (Drupal I'm looking at you).
Often these servers will load 128+GB of data into RAM (crazy, huh?) and even if you have 1GB/sec it's still two minutes for the server to start up.