Ask HN: Best document storage solution in 2023 We have a application which requires a document store. This need to hold up to 10 million documents and be accessible via APIs to retrieve documents to display in our application. We are considering everything from Dropbox-type solutions to blob storage in GCP. What kind of document storage solutions are people using in 2023 to meet this use case? |
Wasabi gives you a free bandwidth allowance equal to the number of bytes stored per month. When I use up most of that, I start pulling from B2. And of course, if one of them is down, I pull from the other.
It's more time up front to build instead of just relying completely on GCP/Azure/AWS. But I don't have to worry as much about spontaneous account terminations destroying my business.
We use S3 + Cloudfront for documents that we want to be quickly accessed by our customers. We use SFTP for our internal docs when we don't care that much about availability and speed.
Currently working on the indexing feature to index the contents of the files too.
In Open-source Ceph and Minio are common. Garage is newer and has good potential too and it has a simpler design.
https://ceph.com/en/ https://min.io/ https://garagehq.deuxfleurs.fr/
If you are already using a particular cloud provider for the web application that will be accessing these documents, you should use their blob storage unless you have a compelling reason not to. Communication between a cloud provider's compute and storage services within the same region will have minimal latency.
The files are stored in S3, with customizable metadata storage in DynamoDB. As the system is designed to run on AWS Serverless and Managed Services, the majority of the cost will come from S3 storage fees.