Show HN: EloqDoc: MongoDB-compatible doc DB with object storage as first citizen

Show HN: EloqDoc: MongoDB-compatible doc DB with object storage as first citizen(github.com)

46 points by iamlintaoz 211 days ago | 18 comments

We're excited to share EloqDoc, a new open source document database built on top of Data Substrate. EloqDoc is designed around the principle of treating object storage (like S3) as a first-class citizen for durability and cost efficiency. If you love the flexibility of MongoDB's document model but are struggling with scaling, cost, and consistency due to its coupled architecture, EloqDoc is for you. It’s built to solve MongoDB's inherent infrastructure challenges while remaining fully compatible with existing MongoDB clients and drivers.

Key Features:

1. Object Storage as First Citizen: Uses object storage for primary durability, leveraging local NVMe caching to achieve both lower cost and higher performance than using block-level storage (e.g. EBS).

2. Decoupled Compute & Storage: Scale your compute/QPS independently of your storage capacity, or vice-versa, without data movement.

3. True ACID Transactions: Delivers full ACID compliance with especially fast distributed transactions—consistency without compromise.

4. Native Distribution & Multi-Writer: It's a natively distributed database, eliminating complex manual sharding routers (like mongos) and supporting true Multi-Writer scalability.

Check it out: https://www.github.com/eloqdata/eloqdoc

We welcome any feedback, critique, or questions on the EloqDoc!

fluxkernel 211 days ago |

There is a discussion on Reddit for a 5TB data volume but very low throughput, I think Object Storage based document database would help in this case. https://www.reddit.com/r/mongodb/comments/1o41dta/is_there_a...

iamlintaoz 211 days ago | |

Yes, this is exactly the typical use case where EloqDoc shines.

Our auto-tiering feature allows you to manage a massive database (e.g., 5TB+) primarily on cheap, durable object storage (like S3), which is already built for cross-AZ replication and is up to 3x cheaper than traditional cloud block storage (EBS).

We use local NVMe SSDs (200K+ IOPS) purely as a high-performance cache layer to accelerate read access. Hot and cold data are automatically swapped across memory, SSD, and object storage tiers based on access frequency, ensuring high performance (up to 100x faster than archive solutions)

hodr_super 211 days ago | |

Mongo Atlas also supports using archive database to store infrequently accessed cold data. It’s also object storage based.

fluxkernel 211 days ago | | |

Typically a separate archive database is not desired choice. You need to specify the rule how old the data is move from production database to archive database. Also the archive database is read only and slow, you can not achieve transaction between production database and archive database.

akshayshah 210 days ago |

Very cool! Using “object storage for primary durability” seems difficult for any OLTP workload that’s latency-sensitive - there’s a fundamental tradeoff between larger batch sizes to control write costs and smaller batches to reduce latency. This hurts OLTP workloads especially badly because applications often make multiple small writes to serve a single user-facing request. How does EloqKV navigate this tradeoff?

Also, I’d love to see:

- A benchmark that digs into latency, throughput, and cost for a single workload. Most of the benchmarks I saw are throughput-only.

- Some explanation of the “patented 1PC protocol.” Your website [1] suggests that you treat single EBS volumes as high-durability, replicated storage, which seems unusual to me - apart from io2 volumes, EBS is designed for less than 3 nines of durability [2].

[1]: http://www.eloqdata.com/blog/2025/07/15/data-substrate-detai...

[2]: https://aws.amazon.com/ebs/features/

iamlintaoz 209 days ago | |

These are great questions. I appreciate you carefully reading through the documents. For the first question, we have detailed benchmark on EloqKV, with the same architecture (but with Redis API) in our blog, and we will soon publish more about the performance characteristics of EloqDoc. Overall, we achieve about the same performance as using local NVME SSD, even when we use S3 as the primary storage, and the performance often exceed the original database implementation (in the case of EloqDoc, original MongoDB).

As for the durability part, our key innovations is to split state into 3 parts: in memory, in WAL, and in data storage. We use a small EBS volume for WAL, and storage is in S3. So, durability is guaranteed by [Storage AND (WAL OR Mem)). Unless Storage (S3) fails, or Both WAL (i.e. EBS lost) AND Mem fail (i.e. node crash), persistence is guaranteed. You can see the explanation in [1]

[1] https://www.eloqdata.com/blog/2025/07/16/data-substrate-bene...

matharmin 210 days ago |

The project looks great! Object storage is often so much better in terms of cost efficiency than a database on EBS. It's often 10-20x more expensive for EBS after taking into account that you need 3x replicas for a typical MongoDB deployment, and need to over-provision the storage. And being able to scale compute independently from storage is great.

The biggest things I'm missing from the docs (checked the github page and the site) is seeing what MongoDB features are supported or not. I've worked with Azure CosmosDB before, and even though it claims MongoDB compatibility, it has many compatibility issues as soon as you have more than a basic CRUD application. Some examples include proper ChangeStream support, partial index support, multi-key index support, set of supported aggregation pipeline operations, tailable cursor support, snapshot queries.

Another thing that's not clear: What does multi-master/multi-write mean in practice? What happens if you write to the same data at the same time on different nodes?

iamlintaoz 210 days ago | |

That's exactly the reason. S3 is better in almost all aspects compared with EBS, except the performance part, and I am glad that our Data Substrate technology solved this issue gracefully [1].

As for the compatibility, we are leveraging some of the code from 4.03 version (the last AGPL version), and we have a very good compatibility (we will show some results in later blog posts). As I mentioned in another reply post, the Mongo APIs are reasonably stable over the last few years, only seeing very minor changes. Most of the later versions improved upon performance and transaction supports, which we support natively with our underlying data substrate technologies. Still, if you have any specific API that you feel is needed, we'd be happy to implement and we welcome community contributions.

Multi-master/multi-writer means it is a fully distributed database. Of course you can run it in single node configurations and get all the single node benefits, but if deployed in a cluster, you do not need to worry about which node to write to, or how data are sharded. If you writes potentially can cause conflicts (i.e. write to the same data at the same time on different nodes), the concurrency-control will handle that for you. In fact, you will encounter the same issue even in a single node configuration, since a single node is still multi-threaded.

[1] https://www.eloqdata.com/blog/2025/07/16/data-substrate-bene...

iamlintaoz 210 days ago |

For full transparency, I am the CEO of EloqData and I am happy to answer any questions you have. EloqDoc has been open-sourced for awhile, and we are glad to announce that the current code is stable for you to test on your workload.

lioeters 209 days ago |

Oh how interesting, there are related database projects built on the same foundation library/tech called Data Substrate.

- EloqSQL - MySQL-compatible, high performance, elastic, distributed SQL database https://github.com/eloqdata/eloqsql

- EloqKV - Redis/Valkey Compatible Distributed Transactional Key-Value Store https://github.com/eloqdata/eloqkv

the_precipitate 211 days ago |

FerretDB and DocumentDB seem to be the only other two Mongo-compatible open source database implementations. Both are based on PostgreSQL.

iamlintaoz 211 days ago | |

That’s correct. FerretDB and DocumentDB both pursue PostgreSQL-based document models.

EloqDoc follows a fundamentally different architectural path. Instead of executing in Postgres, we leverage the existing MongoDB parser and executor to ensure maximum compatibility. Our main contribution is to replace the single-writer WiredTiger storage engine entirely with Data Substrate.

The Data Substrate is an abstract layer built specifically to handle distributed database fundamentals: scalable buffer pooling, concurrency control, durability, elasticity, and fault tolerance. You can read more: https://www.eloqdata.com/blog/2025/07/14/technology

This architectural choice is what enables EloqDoc to deliver features that Postgres-based solutions cannot easily match, including: full compatibility, native Multi-Writer capability, Object Storage First and extremely low-latency distributed transactions.

matharmin 210 days ago | |

FoundationDB also has a Mongo-compatible document layer, but it seems like the last release was 6 years ago, so probably doesn't count anymore.

PeterZaitsev 211 days ago |

I see the license is AGPLv3 while current MongoDB versions as SSPL - does it means you took the MongoDB Parser circa 2018 as a base ?

iamlintaoz 211 days ago | |

That is right. We leverage some of the AGPL MongoDB code for the parser, as indicated by the License. Our own code can be licensed differently, see a previous discussion on Hacker News [1]. The Mongo API are reasonably stable over the last few years, only seeing very minor changes. Most of the later versions improved upon performance and transactions, which we support natively with our underlying technologies. Still, if you have any specific API that you feel is needed, we'd be happy to implement and we welcome community contributions.

[1] https://news.ycombinator.com/item?id=44937978

freakynit 210 days ago |

Nice one..

Btw, your login/signup page auth is broken. Isn't moving ahead after google auth. Fyi, I use firefox in strict privacy mode.

iamlintaoz 210 days ago | |

Thank you so much for reporting this! My local testing on Chrome privacy mode was fine, but we will definitely prioritize checking and fixing the issue in Firefox's Strict Privacy Mode.

freakynit 210 days ago | | |

Cool.. thanks..