Any chance of writing up a case study in detail to share your experience and best practice, especially the comparison with other DBs?

Could you add some details about your scale?

One of my constant pain point with all distributed data stores is that it's really hard to find out how they behave if something breaks. Be it the network, local storage and so on. How do I find out what's wrong? Are there guides on how to fix a problem? What happens if I lost more nodes than required to automatically recover? How does backup and restore work? Any estimates on how long a restore will take? Are there failure modes that I should monitor for that might be non-obvious? This is mostly the operations side, but personally I would never use something I don't understand enough to have a good feeling of how the system works beneath the shiny surface.

And of course there's the application side: With SQL and EXPLAIN, I can usually see bottlenecks. I have a latent fear that performance with distributed systems suddenly tanks if some structure is suddenly split across nodes for example.

- transaction size and duration limitations. I can almost understand the limitation on large write transactions, but the same size limitation applies to read transactions. If you’re doing a large range read, you may not know whether your range will reach the 10MB limit, and thus raise an exception.

- the storage backends seem less impressive than the marketing leads you to believe. The default memory backend is obviously too limited to use in production, and the “ssd” backend turns out just to be built on top of the Btree code from SQLite. Besides that, the documentation warns against using the ssd backend on macOS. Isn’t that a bit strange, considering who owns foundationdb??

- while testing, I found that it was impossible to shrink a cluster. If you add a second storage node just to test that the distributed stuff works correctly, you can’t reduce it back to a single node without destroying the entire database and starting over. If it’s possible to run everything on one node, it should be possible to shrink a cluster back to a single node.

- the storage backends have a crazy amount of write amplification (something like 3x, according to the docs). The foundationdb folks should focus on improving the underlying storage, for instance by building on lmdb or RocksDB or something. For my toy app, I abstracted my data access to use either lmdb (for local testing) or foundationdb (for production), but I ultimately ended up just using lmdb because I didn’t want to deal with fdb’s limitations and operational unknowns.

- another weird fdb limitation: the best single threaded latency you’ll get is supposedly around 1ms for small reads. The docs suggest you can achieve much better performance by scaling the cluster and number of clients. That may be true, but some applications may want high single-threaded performance. (Something like lmdb can achieve tens of thousands of reads per second)

We are actually spend two weeks rewriting our code from SQL to FoundationDB (we have built our own simple ORM).

One of the huge problems is a lack of consistency constraints. Since during development data make became broken in various ways (and we have done several times). And right now there are no way to implement simplest constraints on database level.

My blocker is that the documentation is confusing and not very straight forward. The tutorials are not very verbose and the code (at least for python) is not written in a way that is easy for someone to understand.

I'd be very interested in usage for time-series, but it looks like to get a working example up I'd need to fully parse the documentation and tutorials in order to do it as there is no "Let's build a Timeseries Database on FoundationDB from scratch"

Deployement strategies guide from single node cluster to single datacenter to multi region. And how version upgrades happens

+1 for a SQL layer/interface. Would massively increase utility of FDB.

- Secondary Index

- some way to store ordered records to speed up range query

- comparison & benchmarks of FoundationDB vs other distributed DB like TiDB, CockroachDB, ScyllaDB, etc

We use Postgresql heavily and it's not really a viable replacement. Even if there were an SQL layer, I doubt it would be fully featured enough for our needs. We may evaluate FDB as a distributed cache, but haven't had the time.

I am considering FoundationDB for a SaaS project. Here are some questions I have:

1. What is the recommended way to filter on multiple indexes?

2. What is the recommended way to filter on one or more indexes and sort on another or more index(es)

3. If I use FoundationDB as my main datastore, how should I implement full text search?

4. From what I understand, FoundationDB stores keys sorted globally. How does it handle hot spots where a range of keys that is very frequently accessed is on one machine that gets overloaded?

Edit 5. How do you do backups?

To me it's several things: 1. I don't know how stable foundationdb actually is. You say it's tested and very performant but simply starting fdbmonitor on localhost with two processes and then starting fdbcli, it takes 5.6 seconds to get a connection. Typing 'status' takes even longer (before I have a database configured). Why does it take so long for me to get back an error? What could possibly justify that kind of delay? I know I'm saying the word 'should' a lot of times but in my mind it should be practically instant, especially if there is no network trip involved and you know all the servers that I configured. Secondly, the errors are scary to me. "The coordinator(s) have no record of this database. Either the coordinator addresses are incorrect, the coordination state on those machines is missing, or no database has been created." The error "The database is unavailable" is equally scary. I don't even have a database yet but sometimes I feel like I did create one and then I imagine myself getting that error in production because of some simple or not so simple mistake and I think:

What would I even be doing if the coordinator state is missing? That sounds like a critical internal issue to me, is this really something I need to find out and if it is, why does the error not tell me how to check this or repair it?

Another thing: Right after I create a database with 'configure new single memory', I get in 'status': " Moving data - unknown (initializing) Sum of key-value sizes - unknown " Is this really such an edge case that the database cannot give me meaningful information on what is going on with my data right now? That is very scary to me. It should be able to tell me what is going on with the data, this is the simplest possible moment. I just created a database 5 seconds earlier and I run a status command. Now the database is apparently in an unknown state. What I would like to get is a realtime progress bar of where data is moving and why, with percentage numbers indicating the progress and if there is any error it should actually tell me the location of the log file.

The other big question mark is: I dont know if I can truly configure everything correctly so I actually get the performance that is advertised, there seem to be a lot of things I need to consider and configure exactly the right way in order to get the correct performance as I've read on the forums, from the number of processes to start, to all kinds of other configuration issues. Somebody actually made a guide and I'm thankful for this but in my opinion I should not need to worry about any of this, foundationdb should scan what CPUs/disks I have and suggest the optimal solution/configuration. These are simple rules, shouldnt they be automatically done? Imagine how many people are running foundationdb suboptimally just because they didnt know that the default naive way of starting a few processes and letting them do some work is horribly wrong.

The third issue is: What matters is how the performance will look after I did all my fancy custom layers on top and when I had to emulate all the stuff that was builtin by mongodb and whether this thing is actually as reliable and performant as it says it is. When I look at the issues I enumerated in the beginning, it really doesn't feel good to me.

> - monitoring,

What is the state of production monitoring of FoundationDB?

It feels like the 80% Lower Receiver of databases, so more layers like an SQL layer or something similarly powerful would be ideal.

SQL interface

Trust. Apple already killed it once, just after they bought it. How am I supposed to trust that they wouldn't do so again?

A SQL interface might lessen the impact of that somewhat, but then I might as well just use Postgres from the start.

Only tangentially related, but are there any public Postgres benchmarks for systems running high-CPU, high-memory, p4800x optane drives?

A mate of mine is ex-apple and said they use it for icloud.

FoundationDB and Spanner both offer external consistency. Spanner does this through synchronized clocks. FoundationDB has a similar clock called TimeKeeper, which is not a clock per se but a counter which advances approximately 1M times per second. Transactions are ordered based on this timestamp.

100ms+!

Ouch. Is that just large amounts of data? Are you guys using pg.native?