RedisRaft(github.com) |
I am guessing maybe it's Apple's corporate secrecy that is the issue. Apple likely has a massive deployment of this tech.
Secondary indexes in "distributed strongly consistent" systems is what ruins performance: because each index is +1 write to another "index table" (... +1 raft quorum check!).
I don't think FoundationDB has "secondary indexes" to begin with, so one may never run into the +1 write per index issue.. it's just a layer on top of TiKV (equivalent to RocksDB in CockroachDB).
Calling FoundationDB "just a layer on top of TiKV" is… well… :-)
Can't secondary indexes just be implemented in your state machine?
I.e. your state machine handles the insert and also writes a secondary index at the same time. Every state machine could do this identically off the log.
Will indeed be interesting to see the analysis once it becomes stable.
> RedisRaft is still being developed and is not yet ready for any real production use. Please do not use it for any mission critical purpose at this time.
This is like saying laws around seatbelt wearing are a fabrication because some people ignore them..
[1] https://www.zdnet.com/article/redis-labs-unveils-redis-datab...
It has been running in a intercontinental production environment with 100% read uptime since 2017.
It's 2000 lines of code: http://root.rupy.se (this test env. has 3 nodes: fem, six and sju)
Average is 4.8 millisec and 0.5 millisec respectively.
But those are just the JVM doing it's thing.
The numbers I like are these: 190.2 158 331
200ms global save average, 158ms min and 331ms max from europe to east+west US and Asia. Without fault, so consistent, much of that can be attributed to AWS improving so much over the years.
As to why the load/save are slower it's because the complete global roundtrip stats I only have for registers which are rare now so they don't hit the GC I'm guessing.
The thing I'm most proud of is async-to-async meaning the system will saturate all cores (without io-wait) on all machines without problems... it just keeps solving the problem at 100% efficiency, no memory leaks and 5 years uptime without any crashes.
Slowing down is the worst case, and if that is a problem just upgrade instance type, no uptime then though.
What a weird notation. When N=3, a cluster may lose up to 1 node, I don't know how that matches this formula.
https://decentralizedthoughts.github.io/2020-12-12-raft-live...
MemoryDB seems to have a very similar architecture to that of AWS Aurora, which separates a storage layer and compute nodes and consistency is implemented not by communicating between compute nodes but by offering a consistent distributed storage layer. This architecture usually don't have a multi-node strong consistency by itself and can have replicas.
This means that in MemoryDB only the primary node is strongly consistent but the replica nodes don't.
Instead, in my experience, those kinds of AWS offerings have less operational headaches because the storage remains safe even the primary node fails and you don't need to worry about managing distributed nodes.
Edit: add pros
This comes at a cost though and writes are slower than traditional redis.
More details: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml...
Code being ready for production has nothing to do with code being shared privately nor publicly.
Secondary indexes are implemented as an additional smaller table in most databases (Index -> Primary Key), which requires a write, too.
Don't take my word for it- run some benchmarks yourself to see the performance cliffs appear in these "distributed strongly consistent" databases when you have a handful of indexes.
It's why many require 10x the hardware to scale past the performance of a single node postgres / mysql.
> Calling FoundationDB "just a layer on top of TiKV" is… well… :-)
FoundationDB recommends running TiDB ontop to get an SQL layer...
You might be mixing up projects here - TiDB/TiKV are completely separate projects from FoundationDB. I don't think FoundationDB has a supported SQL layer.
> Don't take my word for it- run some benchmarks yourself to see the performance cliffs appear in these "distributed strongly consistent" databases when you have a handful of indexes.
I'm not saying secondary indexes aren't a performance cliff. I'm saying I can't think of a reason they must require additional quorum on top of the initial message that would require a secondary index update.
Since you said:
> Secondary indexes in "distributed strongly consistent" systems is what ruins performance: because each index is +1 write to another "index table" (... +1 raft quorum check!).
This depends on the specifics of how your database works, of course. I’m not sure this would involve multiple consensus quorums in FDB for instance, given that I believe it instead relies on a centralised timestamp service.
There are some well supported document interfaces like the official record layer or Tigris though.