I just want simple S3(blog.feld.me) |
I just want simple S3(blog.feld.me) |
Disclaimer: I work at HF
"firt commit" [0] was less than a month ago and added 51k lines in a single commit.
0: https://github.com/floci-io/floci/commit/61433f59ab995e9eaeb...
Tres bon!
You failed to answer why you even need s3... Why not a filesystem? Full stop. The entire point of s3 is distributed.
10 users with simple needs = complex software.
If they want it for local dev work, that's pretty different from wanting a high-performance air gapped object store without rewriting clients.
They seem to know what they're doing (having complained about a methodology problem in MinIO), and yet don't personally want to throw their hat in the ring not maybe pay anyone...
Context matters!
> I just want S3. My needs are pretty basic. I don't need to scale out. I don't need replication. I just need something that can do S3 and is reliable and not slow.
What does reliable mean, without replication, and no mention of backups?
I just need something that can do S3 and is reliable and not slow.
I'm a simple man, I just need edge delivered cdn content that never fails and responds within 20ms.
If you split the interaction API out to an interface detailing actual program interaction with the service, then write an s3 backend and an FS backend. Then you could plug in any backend as desired and write agnostic application code.
Personally I end up there anyways testing and specifying the third party failure modes.
That repo is a fork of this project: https://github.com/johannesboyne/gofakes3
They bill it as being for testing, but it works great if all you want is a no-fuss S3-compatible API on top of a filesystem. I've run it on my NAS for a few years now to provide a much faster transfer protocol compared to SMB.
It doesn't need to care about POSIX mess but there is whole swathes of features many implementations miss or are incomplete, both on frontend side (serving files with right headers, or with right authentication) and backend (user/policy management, legal hold, versioning etc.)
It gets even more messy when migrating, for example migrating your backups to garagefs will lose you versioning, which means that if your S3 secret used to write backups gets compromised, your backups are gone vs on implementation that supports versioning you can just rollback.
Similarly with password, some will give you secret and login but won't allow setting your own so you'd have to re-key every device using it, some will allow import, but only in certain format so you can restore from backup, bot not migrate from other software.
QS3 (Quite Simple Storage Service) for the barebones. Bucket/Object CRUD. Maybe: Multipart Uploads. Presigned URLs.
S3 for Object Tagging, Access Control Lists, etc.
S3E (enterprise? extended? elaborate?) for Object Lock & Retention (WORM compliance, Legal Holds), Event Notifications and so on.
* S4: Stupid/Silly Simple Storage Service
* S3: Simple Storage Service
* S2: Storage Service
SeaweedFS and RustFS both look nice though, last I checked Zenko was kinda abandoned.
Also pretty simple when I can launch a container of whatever I need (albeit bootstrapping, e.g. creating buckets and access keys SHOULD probably done with env variables the same way how you can initialize a MySQL/MariaDB/PostgreSQL instance in containers) and don't have to worry too much about installing or running stuff direcetly on my system. As for unsupported features - I just don't do the things that aren't supported.
It strikes me as a classic case of "we need all the interested people to pull in one project, not each start their own". AI may have made this worse then ever.
I don't have a horse in this game, but have had pretty reasonable results using SeaweedFS for GitHub Actions caching. RustFS is on my list to test next, and a team mate is quite keen on Garage.
S3 is simple for the users, not the operators. For replicating something like S3 you need to manage a lot of parts and take a lot of decisions. The design space is huge:
Replication: RAID, distributed copies, distributed erasure codes...
Coordination: centralized, centralized with backup, decentralized, logic in client...
How to handle huge files: nope, client concats them, a coordinator node concats them...
How will be the network: local networking, wan, a mix. Slow or fast?
Nature of storage: 24/7 or sporadically connected.
How to handle network partitions, pick CAP sides...
Just for instance: network topology. In your own DC you may say each connection has the same cost. In AWS you may want connections to stay in the same AZ, use certain IPs for certain source-destination to leverage cheaper prices and so on...
I’ve been at a couple companies where somebody tried putting an S3 interface in front of an NFS cluster. In practice, the semantics of S3 and NFS are different enough that I’ve had to then deal with software failures. Software designed to work with S3 is designed to work with S3 semantics and S3 performance. Hook it up to an S3 API on what is otherwise an NFS server and you can get problems.
“You can get replication with RAID” is technically true, but it’s just not good enough in most NFS systems. S3 style replication keeps files available in spite of multiple node failures.
The problems I’m talking about arise because when you use an S3-compatible API on your NFS system, it’s often true that you’re rolling the dice with three different vendors—you have the storage appliance vendor, you have the vendor for the software talking to S3, and you have Amazon who wrote the S3 client libraries. It’s kind of a nightmare of compatibility problems in my experience. Amazon changes how the S3 client library works, the change wasn’t tested against the storage vendor’s implementation, and boom, things stop working. But your first call is to the application vendor, and they are completely unfamiliar with your storage appliance. :-(
NFS is just an interface. At the end of the day it's on top of an FS. It's entirely possible and sometimes done in practice to replicate the underlying store served by NFS. As you would expect there are several means of doing this from the simple to the truly "high-availability."
It's a little like asking why you'd use SQL.
And SQL is also very important. And yet, if somebody said "I need to store data, but it's not relational, and I just need 1000 rows, what's the best SQL solution," I would still ask why exactly they needed SQL. The might have a good reason (for example, SQLite can be a weirdly good way to persist application data), but I don't know it yet. That's why I asked.
And I want things like backup, replication, scaling, etc. to be generic.
I wrote a git library implementation that uses S3 to store repositories for example.
- MinIO [3] is somewhat deprecated and not really open source anymore. Its future is unsure.
- GarageHQ [4] looks pretty great and I wished I could have used this, but it is not yet feature-complete with S3 protocol and specifically missing the versioning feature (I reported this [5])
- Zenko Scality works out of the box; it is a bit too "big" for my context (aimed at thousands of parallel users) and uses 500MB memory; but it does the job for now.
I posted my compose here [6]. Since then (~months ago), it works really well and I am happy with Zenko Scality S3.
[1]: https://github.com/gristlabs/grist-core
[2]: https://github.com/scality/cloudserver
[3]: https://www.min.io/
[4]: https://garagehq.deuxfleurs.fr/
[5]: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/166
[6]: https://github.com/scality/Zenko/discussions/1779#discussioncomment-15869532last year they had a security vulnerability where they allowed a hardcoded "rustfs rpc" token to bypass all authentication [0]
and even worse, if you read the resulting reddit thread [1] someone tracked down the culprit commits - it was introduced in July [2] and not even reviewed by another human before being merged.
then the fix 6 months later [3] mentions fixing a different security vulnerability, and seemingly only fixed the hardcoded token vulnerability by accident. that PR was also only reviewed by an LLM, not a human.
0: https://github.com/rustfs/rustfs/security/advisories/GHSA-h9...
1: https://www.reddit.com/r/selfhosted/comments/1q432iz/update_...
Edit: Minio is written in Go, and is AGPL3... fork it (publicly), strip out the parts you don't want, run it locally.
Then I decided to run the Ceph S3 test suite against it. So many issues. Think it passed 3 tests on first run out of I think about a hundred. Took another couple of hours to get it to a state that is even vaguely passable for non production use.
Got something vaguely workable but even after many hours I can’t say I super trust it
> implementation can't be that complicated.
S3 has a fair bit of bloat in spec that nobody seems to use and it’s not clearly delineated which is core and what’s ahem optional. I ended up relying on the LLM to figure out what’s core and that ended up missing stuff too so that needed a couple of feature iterations too
I also had quite a hard time to chose one for my homelab too. Settled on garage too but why is it that complicated ? I just want a "FS api" for remote storage in my app. Over http so that i dont have to do weird custom setup just to have a network share and handle everything manually
``` docker exec $CONTAINER $BIN key import --yes "$S3_ACCESS_KEY_ID" "$S3_SECRET_ACCESS_KEY" ```
Most notably, the PutObject operation (which had always been a pain in the ass on HDD with MinIO) is performing well now, even with many small objects.
There is a natural synergy in the gateway storing its metadata in xattrs and using ZFS special VDEV with dnode_size=auto to store the entire ZFS+S3 metadata on fast media.
The latency impacts of the gateway itself can be further minimized by running multiple instances of the gateway pinned to CPU cores, all behind a HAProxy load-balancer communicating with them over UDS.
So I too just want simple S3. Minio used to be the no-brainer. I'll checkout RustFS as well.
It does not sound hard (although it is hard for me!). It sounds like it should be some LinuxServer io container. Doesn't it? At this point S3 is just as standard as WebDav or anything right?
https://canonical-microceph.readthedocs-hosted.com/stable/tu...
Impressive as hell software and I am so glad to have it. But man! The insistence on mountains of ram per TB, on massive IO is intimidating.
Its also not that useful even if you have enough machines to run it properly.
NVME and zfs is fast enough for virtually anything now. With snapshot and snapshot sending you get decent backups for half the hardware cost of ceph.
Can’t say I super trust it so will probably roll it out only for low stakes stuff. It does pass the ceph S3 test suite though and is rust so in theory somewhat safe-ish…maybe.
You either are doomed to reimplement and rediscover the complexity on your own, or you change your requirements to fit a narrower problem domain to avoid (some of) the complexity.
If they just want webfile interface then a webserver with simple auth and webDAV would work more than well enough.
The problem is that they then go onto talk about lots of projects that all have posix interfaces. Which is slap bang into shared filesystem land.
S3 is not a filesystem, and nothing shows that more than when you use it as an object store _for_ a filesystem.
Depending on the access requirements, if you're doing local to local, then NFSv4 is probably more than enough. Unless you care about file locking (unlucky, you're in shit now)
Every month there's a post of "I just want a simple S3 server" and every single one of them has a different definition of "simple". The moment any project overlaps between the use cases of two "simple S3" projects, they're no longer "simple" enough.
That's probably why hosted S3-like services will exist even if writing "simple" S3 servers is so easy. Everyone has a different opinion of what basic S3 usage is like and only the large providers/startups with business licensing can afford to set up a system that supports all of them.
And every few weeks in the cooking subreddit we get a new person talking about a new soup they made. Just think if we put all 1000 of those cooks in one kitchen with one pot, we'd end up with the best soup in the world.
Anyway, we already have "the one" project everyone can coalesce on, we have CephFS. If all the redditors actually hopped into one project, it would end up as an even more complex difficult to manage mess I believe.
Or like lodash custom builds.
I mean you can, it would simplfy the locking somewhat.
But if you are doing file sharing for apps inside a network you manage, just use NFS, and maybe worry about the locking later.
Like it or not, S3 has become the de facto API for object storage for many developers. From the operations side of things, managing files is easier and already taken care of by your storage solution, be it a SAN, NAS or something entirely different. Being able to backup and manage whatever is stored in S3 with your existing setup is direct saving.
If you actually use a large subset of S3s features this might not be good solution, but in my experience you have a few buckets and a few limited ACLs and that's it.
- Lets me deploy stateless containers easily
- Let’s me leverage the NAS for local redundancy and a more centralized place to do backups
- When a project grows it’s easy to promote it to use a hosted S3
- Local S3 becomes a target for Litestream and Restic
- Developing against the local fs and then handling file storage is a huge friction, unless I’m using something like Rails that already has a good abstraction
Also think there is an abstraction mismatch with the object stores that store the objects with a 1:1 mapping into the file system. Obvious issues are that you only get good listing performance with '/' as delimiter and things like "keys with length up to 1024 bytes" break.
I'm not saying that there weren't reasons to add these functions to the protocol, but if your aim is minimalism, then you can do _much_ better, and I think there is a real benefit in having a bare bones protocol that anyone could implement in many contexts.
That’s the entire point: my domain code ‘stores’ something, my implementation decides if that is S3 or a fixed kind of error or writes to a disc.
If my domain uses a third party app that needs an S3 API, and we see not pointing it at S3, then I am writing an s3 API to test that dependency and ensure the third party code works regardless. Normally I’d call that “their problem” and call their service with the same wrapper as above.
That test matrix uncovered that post policies were only checked for exsitence and a valid signature, not if the request actually conforms to the signed policy. That was an arbitrary object write resulting in CVE-2026-27607 [2].
In the very first issue for this bug [3], it seemed that the authors of the S3 implementation didn't know the difference between the content-length of GetObject and content-length-range of a PostObject. That was kind of a bummer and leads me to advise all my friends not to use rustfs, though I like what they are doing in principal (building a Minio alternative).
[1]: https://github.com/nikeee/lean-s3 [2]: https://github.com/rustfs/rustfs/security/advisories/GHSA-w5... [3]: https://github.com/rustfs/rustfs/issues/984
* create rustfs user * run the rustfs from root via systemd, but with bunch of privileges removed * write logs into /var/logs/ instead of /var/log
Looks like someone told some LLM to make docs about running it as service and never looked at output
You can't exactly do "this client is allowed to download this one specific file for the next 24 hours (but not use it like their personal CDN and share the link with everyone)" or "this client is allowed to upload a single image file up to 10MB (but not upload a 100GB copy of Friends)" with basic bearer tokens.
There are lots of benefits to this over the bearer token approach that many take. Binding the authentication to the specific request is valuable regardless of TLS.