Of course, despite working on this for a week I only now discovered this... dm_linear is an easier way than mdadm to concatenate the loopback devices into a single device. Setting up the table input to `dmsetup create`'s stdin is more complicated than just `mdadm --build ... /dev/loop1{0000..9999}`, but it's all scripted anyway so it doesn't matter. And `mdadm --stop` blocks for multiple minutes for some unexplained reason, whereas `dmcreate remove` is almost instantaneous.

One caveat is that my 1MB (actually 999936B) block devices have 1953 sectors (999936B / 512B) but mdadm had silently only used 1920 sectors from each. In my first attempt at replacing mdadm with dm_linear I used 1953 as the number of sectors, which led to garbage when decrypted with dm_crypt. I discovered mdadm's behavior by inspecting the first two loopback devices and the RAID device in xxd. Using 1920 as the number of sectors fixed that, though I'll probably just nuke the LUKS partition and rebuild it on top of dm_linear with 1953 sectors each.

What a coincidence, I just recently did something similar.

Did you run into any problems with discard/zeroing/trim support?

This was a problem with sshfs — I can’t change the version/settings on the other side, and files seemed to simply grow and become more fragmented.

I suspected WebDAV and Samba might have had been the solution but never looked into it since sshfs is so solid.

FWIW this is similar to Apple's "sparse image bundle" feature, where you can create a disk image that internally is stored in 1MB chunks (the chunk size is probably only customizable via the command line `hdiutil` not the UI). You can encrypt it and put a filesystem on top of it.

Are you using davfs2 to mount the 1MB files from the WebDAV server?

This is a very nice solution.

Wouldn’t you be maintaining your own list of what is in the zip offline at this point?

Because you don't have to allocate any fixed amount up front, and it's pay as you go. At the time when the best storage options you could get were fixed-size hard drives from VPS providers, this was a big change, especially on both the "very small" and "very large" ends of the spectrum. It has always spoken HTTP with a relatively straightforward request-signing scheme for security, so integration at the basic levels is very easy -- you can have signed GET requests, written by hand, working in 20 minutes. The parallel throughput (on AWS, at least) is more than good enough for the vast, vast majority of apps assuming they actually design with it in mind a little. Latency could improve (especially externally) but realistically you can just put an HTTP caching layer of some sort in front to mitigate that and that's exactly what everybody does.

Ceph was also released many years after S3 was released. And I've never seen a highly performant 9P implementation come anywhere close to even third party S3 implementations. There was nothing for Amazon to copy. That's why everyone else copied Amazon, instead.

It's not the most insanely hyper-optimized thing from the user POV (HTTP, etc) and in the past some semantics were pretty underspecified e.g. before full consistency guarantees several years ago, you only got "read your writes" and that's it. But it's not that hard to see why it's popular, IMO, given the historical context and use cases. It's hard to beat in the average case for both ease of use and commitment.

When S3 was released the Internet was very different. Two of the things that stood out were:

1. It offered a resilient key/object store over HTTP.

2. By the standards of the day for bandwidth and storage it was (and to a certain extent still is) very inexpensive.

Since then much of AWS has been built on the foundation of S3 and so its importance has changed from merely being a tool to basically a pervasive dependency of the AWS stack. Also, it very much is designed for objects larger than 1KB and for applications that need durable storage of many, many large objects.

The key benefit, at least according to AWS marketing, is that you don't have to host it yourself.

Simple api

Absurdly cheap storage

Extremely HA

Absurdly durable

Effectively unlimited bandwidth

Effectively unbounded storage without reservation or other management

Everything supports its api

It’s not a file system. It’s a blob store. It’s useful for spraying vast amounts of data into it and getting vast amounts of data out of it at any scale. It’s not low latency, it’s not a block store, but it is really cheap and the scaling of bandwidth and storage and concurrency make it possible to build stuff like snowflake that couldn’t be built on Ceph in any reasonable way.

The problem is S3 is just a lexicographically ordered key value store with (what I suspect is) key-range partitions[1] for the key part and Reed-Solomon encoded blobs for the value part. In other words, it’s a glorified NoSQL database with no semantics that you’d typically expect of a file system, and therefore repeated writes are slow because any modification to an object involves writing a new version of the key along with its new object.

[1] https://martinfowler.com/articles/patterns-of-distributed-sy...

S3 is straight HTTP, the most widespread API. It can be directly used on the browser, has libraries in pretty much every language, and can reuse the mountain of available software and frameworks for load-balancing, redirections, auth, distributed storage etc

I think theres an interesting story in software ecosystems where there are two flavors of applications (which coexist) that prefer object stores over filesystems and vice versa. Good reference point for this I think exists in many modern video transcoding infrastructures.

Using something like FSx [1] gives you a performant option for the use cases when the tooling involved prefers filesystem semantics.

[1] https://aws.amazon.com/fsx/lustre/

Here are reasons I'm using S3 in some projects:

1. Cost. It might vary depending on vendor, but generally S3 is much cheaper than block storage, at the same time with some welcome guarantees (like 3 copies).

2. Pay for what you use.

3. Very easy to hand off URL to client rather than creating some kind of file server. Also works with uploads AFAIR.

4. Offloads traffic. Big files often are the main source of traffic on many websites. Using S3 allows to remove that burden. And S3 usually served by multiple servers which further increases speed.

5. Provider-independent. I think that every mature cloud offers S3 API.

I think that there are more reasons. Encryption, multi-region and so on. I didn't use those features. Of course you can implement everything with your own software, but reusing good implementation is a good idea for most projects. You don't rewrite postgres, so you don't rewrite S3.

> In my personal experience with S3 it’s always been super slow.

Numbers? I feel like it's been a while, but my experience was it is in the 50ms latency range. That's fast enough that you can do most things. Your page loads might not be instant, but 50ms is fast enough for a wide range of applications.

The big mistake I see though is a lack of connection pooling: I find code going through the entire TCP connection setup, TLS setup, just for a single request, tearing it all down, and repeating. boto also enouranges some code patterns which result in GET bucket or HEAD object requests which you don't need and can avoid; none of this gives you good latency.

S3 works over HTTP, which means that it is designed to work over the internet.

Other protocols you mentioned, including NFS, does not work well over the internet.

Some of them are exclusively designed to work within the same network, or very sensitive to network latency.

> Forgive the question but I never quite understood the point of S3.

S3 and DynamoDB are essentially a decoupled BigTable; in that both are KV databases: One is used for high performance, small obj workloads; the other for high throughput, large obj workloads.

They have NFS (called EFS), but it's about 10x more expensive.

S3 is slow but at the same time low cost, if you want fast AWS has other alternatives but pricier.

Same with my experience. Not a fan

Bad ideas are very relative.

Some bad ideas work extremely well if they fit your use case, you understand very well the tradeoffs and you’re building safeguards (disaster recovery).

Some other companies try to convince (force?) you into a workflow or into a specific solution. Aws just gives you the tools and some guidance on how to use them best.

Indeed.

There is no reason why the descriptor of objects can’t be updated with a new value that has all of the old chunks and a new one, since S3 doesn’t do deduplication anyway the other chunks may be resized internally with an asynchronous process that gets rid of the excess data corresponding to the now overridden chunk.

ForkLift also lets you mount S3 as a drive. https://binarynights.com

Or https://clone.org ?

FUSE makes everything worse, not better. The Unix file API is awful in general, and a terrible mismatch for key-values storage systems.