http://web.archive.org/web/20150325003241/http://blog.founda...
I'm not trying to make a comparison between a system I used to work on and one that I frankly know little to nothing about; rather, I'd suggest that building a system like this just isn't enough to be compelling on its own.
As a business it was always an ambitious effort, and I'm not sure what could or should have been done differently. But since then I've used a number of other systems and thought to myself "boy, I wish I had FDB right now."
You're right that distributed consistency is a beginning, not an end. We are painfully aware of all the startups that have died or are dying on this beach.
It's great to be scalable and consistent, but you have to be more than an operationally-better replacement for legacy SQL. That's one reason we built our own query language that plays to modern application development patterns (serverless, functional, change feeds, etc.) instead of the typical slow, never-quite-there, distributed SQL planner.
On mobile so cannot read article
You're doing 3k batches per second with 4 logical writes each, right? So that is at most 3-12k writes per second using the way that every other distributed database benchmark and paper counts.
Or otherwise - if you continue counting writes in this special/misleading way - you'd have to multiply every other distributed db benchmark's performance numbers with a factor of 3-15x to get an apples-to-apples comparison.
The 12k batched writes/sek through what I assume is a paxos variant is still pretty impressive though! Good to get more competition/alternatives for zookeeper & friends!
If you want to include write amplification, then multiply by 6x again to account for the replicated log and the tables themselves.
Counting any kind of "internal write effects" that result from a user write (i.e. write amplification) is obviously done to mislead in the benchmark and does not make it comparable to key-value stores.
12k writes/s is the number of rows that are written from a user perspective. So 12k/s is also the number you have to use when comparing it to key value stores. But of course, comparing Fauna with eventually consistent systems is not a really fair comparison. You don't make it fairer by misleading in your benchmark though.
Also, just because some other vendor posted a misleading benchmark on hn (I don't know if they did) that doesn't make it right or means you should do it. Just call them out on it too.
120,000 writes per second is accurate, talking about actual durable storage (disk) writes. But it's only 3,330 transactions, which should be the number that a user cares about.
I don't have proper data and I'm a bit rusty, but I feel like Cassandra could blow that away if you set similar consistency requirements on the client side (QUORUM on read, same for write?). Am I understanding this correctly, or does Fauna/Calvin give you something functionally better than what C* can do?
YMMV, but we've found the performance of Cassandra writing out similar-sized multi-row atomic batches at QUORUM to be similar in this hardware configuration.
FaunaDB transactions are quite a bit more powerful, as they can span multiple keys, use conditionals and read-modify-write logic, and still resolve with serializable semantics.
How does this algorithm compare to whatever Google Spanner does?
This makes the read performance equivalent to something like Cassandra at CONSISTENCY.ONE, without giving up the cross-partition write linearization of something like Spanner.
I've personally seen a Cassandra ring go to more than 2M ops/sec.
Is this specifically for distributed SQL only? I think there are some scalable SQL systems that don't support sessions either.
Multi-query transactions can be useful, but the FaunaDB query language is functional, rather than declarative like SQL, so composing queries that can do everything you want is usually easier than SQL.
Would you create a single operation that reads one record, checks that it's enough, then adds the amount to another record?
Or maybe you'd first read both accounts, then issue a conditional write operation that makes sure the data hasn't changed before doing the write?
http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
Also a single SSD from 2015 is rated at 120K writes per second:
PM1725: http://www.samsung.com/semiconductor/global/file/insight/201...
The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar...). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.
On the other hand, the Fauna numbers don't seem that impressive to me. On a mid-2011 Macbook Air, I get 2600 transactions per second (read-committed) in PostgreSQL 9.6. Setup is as follows:
CREATE TABLE IF NOT EXISTS foo(a TEXT, b TEXT, c TEXT, d TEXT);
CREATE INDEX IF NOT EXISTS idx_foo_a ON foo(a);
CREATE INDEX IF NOT EXISTS idx_foo_b ON foo(b);
CREATE INDEX IF NOT EXISTS idx_foo_c ON foo(c);
CREATE INDEX IF NOT EXISTS idx_foo_d ON foo(d);
-- prepared statement, the inserted strings are 4 chars wide
INSERT INTO foo(a, b, c, d)
VALUES
($1, $2, $3, $4),
($5, $6, $7, $8),
($9, $10, $11, $12),
($13, $14, $15, $16);
Am I missing something?
If you ran the benchmark yourself, how did you achieve 1m durable writes/sec on a postgres machine/instance? [It's quite an achievement] On what kind of crazy hardware? How large was each write/row? Did you use the postgres network protocol to perform the writes?
Tracking logical writes makes the test comparable to tests of key-value stores that can only update one key at time, which is pretty much every other distributed database.
edit: here's an example in our Scala DSL:
Let {
val amount = 50
val balanceA = Select("data" / "balance", Get(Ref("accountA"))
val balanceB = Select("data" / "balance", Get(Ref("accountB"))
If(Gteq(balanceA, amount),
Do(
Update(Ref("accountA"), Obj("data" -> Obj("balance" -> Subtract(balanceA, amount)))),
Update(Ref("accountB"), Obj("data" -> Obj("balance" -> Add(balanceB, amount)))),
"Transfer Success"
),
"Insufficient Funds"
)
}(BTW, would be nice if I could read your API docs without signing up for an account.)
The docs will eventually be available without an account.
Additionally you have the bottleneck of a single master. Will it be possible to do a quorum read as well, and will every transaction on the master be doing it? Then you are starting to get closer, although many anomalies are still possible.
We tried to replicate a realistic workload rather than just target the best case or worst case performance profile.
And also, by the same logic, replicating a write to three machines counts as three writes when the replication factor is user-defined, right?
(An uninitiated reader would assume you're comitting 120k rows per second from the title, whereas it's "only" 12k rows and "only" 3k actual operations over the wire. Still, 3-12k is pretty impressive)
We should track disk IOPS, though, so you can do an apples-to-apples comparison of how much low-level throughput the database is driving. I believe the instance store disks in the EC2 C3 hardware class can support about ~20K write IOPS each.
Whether Apple's leadership agrees with me is another question. :)
https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va...
My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is obviously usually true.
Nevertheless, I still think this is a bad argument. While it's true that abstractions are rarely costless, they can often be made so cheap that the low-hanging performance fruit is elsewhere. And in particular, cheap enough that they're worth it when you consider all the other benefits that they bring.
When I built a query language and optimizer on top of FoundationDB, my inability to push type information down into the storage engine was about the last thing on my mind. Perhaps someday when I'd made everything else perfect it would've become a big pain (and perhaps someday we would've provided more mechanisms for piercing the various abstractions and providing workload hints), but in the meantime partitioning off all state in the system into a dedicated component that handled it extremely well made the combined distributed systems problem massively more tractable. The increased developer velocity and reduced bugginess in turn meant that I (as a user of the key-value store) could spend scarce engineering resources on other sorts of performance improvements that more than compensated for the theoretical overhead imposed by the abstraction.
I won't claim that a transactional ordered key-value store is the perfect database abstraction for every situation, but it's one that I've found myself missing a great deal since leaving Apple.
But I'm glad to hear that things are going well for you guys. Best of luck, this is a brutal business!
We are happily using them to implement and optimize our local storage on top of LMDB (another awesome database). However, these approaches could be applied to any other key-value database with transactions and lexicographically stored keys.
I have some concerns about CockroachDB on both the performance and the reliability fronts. But I hugely admire what they're trying to do and I've heard that they're rapidly improving in both areas. TiDB is an exciting project that I've heard great things about but have never tried myself. I think it's also relatively immature.
Honestly if I were starting a project right now and had neither FDB nor Spanner available to me, I'd probably try to push Postgres as far as I possibly could before considering anything else.
But if one does need a bit more horizontal scalability, there don't seem to be a lot of options if you also want atomic, transactional updates (though not necessarily strict transaction isolation). I have an app that is conceptually a versioned document store, where each document is the sum of all its "patches"; when you submit a batch of patches, the rule is that these are applied atomically, and that the latest version of document thereafter reflects your patch (optimistic locking and retries take care of serialization and concurrent conflicts). I'm using PostgreSQL right now, which does this beautifully, but with limited scalability. I've looked for a better option, but not come up with anything.
Redis would handle this, but it would work purely thanks to single-threaded; and I don't feel like Redis is safe as a primary data store for anything except caches and such. Cassandra might do it, using atomic batches, although its lack of isolation could be awkward to work around.
I still think many of the arguments in that blog post hold up for non-embedded KV stores. I think you can mitigate a lot by aggressively caching metadata, but eventually you end up moving the SQL engine closer and closer to the storage layer to get performance. And yeah, you end up more monolithic and testing gets harder. Sigh.
Some of this is workload dependent. If you're not touching many rows in your queries and transactions, then you can get away with a lot more. But if you give someone SQL, they're going to want to scan.
I wouldn't mind being proven wrong. Maybe Apple made FDB run SQL at legit speeds. I haven't seen much from public projects that work this way to change my mind yet.
> I won't claim that a transactional ordered key-value store is the perfect database abstraction for every situation, but it's one that I've found myself missing a great deal since leaving Apple.
How does Spanner not satisfy that itch? Not ordered matters?
I was probably unclear in my previous comment. Spanner is great! (And Spanner is ordered). The particular aspect of FDB that I miss is what some of our old customers called "the bottom half of a database" or "a database construction kit". In fact FDB was an awesome modular building block for all kinds of distributed systems, not just databases. We hacked up prototypes for a whole bunch of these but sadly never got around to releasing them.
Spanner is a full-fledged enterprise grade database with opinions about your data model, query language, types, etc. For the vast majority of customers, that's much more useful than what FDB provided. But for me as somebody who enjoys kicking around silly new ideas for distributed systems, it's a bit less fun.
It feels like the building block approach lets you achieve better design and performance for your entire application in the long run. Especially, if you can treat building blocks as blueprints and modify them to fit the task at hand.
What was on your mind? What performance problems did you encounter?