Comdb2 – Bloomberg's distributed RDBMS under Apache 2

Comdb2 – Bloomberg's distributed RDBMS under Apache 2(github.com)

240 points by Callicles 9 years ago | 65 comments

Scaevolus 9 years ago |

Reading through their VLDB paper [1], Comdb2 appears to be a moderately scalable (up to dozens of nodes (?)) RDBMS with a strong emphasis on consistency and availability. Benchmarks show numbers comparable to Percona XtraDB (MySQL with a different storage engine), at ~2,000 writes/s and 2,000,000 reads/s against a 6 node server cluster. High availability and global sequencing is provided by using GPS clocks, similar to Spanner/Truetime.

Schema changes happen lazily, with old rows being rewritten on the next update, and a background job doing bulk rewriting.

Scalability: "While reads scale in a nearly linear manner as cluster size increases, writes have a more conservative level of scaling with an absolute limit. The offloading of portions of work (essentially all the WHERE clause predicate evaluations) across the cluster does help scale writes beyond what a single machine can do. Ultimately, this architecture does saturate on a single machine’s ability to process the low level bplogs."

This doesn't provide the horizontal scaling that Spanner does, CockroachDB aims at, or FoundationDB presumably has.

[1]: http://www.vldb.org/pvldb/vol9/p1377-scotti.pdf

idibidiartists 9 years ago | |

<< or FoundationDB presumably has >>

You mean FaunaDB?

elvinyung 9 years ago | | |

FoundationDB was the proto-CockroachDB (or rather, CockroachDB is essentially a re-attempt at building FoundationDB). It was an early attempt at building a NewSQL database. (NewSQL per se, i.e. not counting parallel databases from the pre-NoSQL age, like Gamma [1], Volcano [2] and Grace [3], which share many of the same design principles.)

FoundationDB was acquhired by Apple, but its failure is generally attributed to a poorly-performing SQL layer: https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

[1] http://pages.cs.wisc.edu/~dewitt/includes/paralleldb/ieee90....

[2] https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b...

[3] https://pdfs.semanticscholar.org/a7f4/e4e6166dc683e7fa7d5b9e...

Scaevolus 9 years ago | | |

No, FoundationDB which was posting some VERY impressive numbers before being acquired by Apple in March 2015.

Here they were doing 15M writes/s on 32 16-core servers, at a rate of 30,000 writes/s/core: http://web.archive.org/web/20150427041746/http://blog.founda...

FaunaDB managed 120,000 writes per second on 15 machines. https://fauna.com/blog/distributed-acid-transaction-performa...

(Yes, not equivalent benchmarks, but that's still a 50x difference in magnitude.)

idibidiartists 9 years ago | | |

Down voted for asking a question? The system is definitely broke in such instances.

macdice 9 years ago |

Some assorted interesting points:

* optimistic concurrency control (sometimes you need to retry, but often the optimism pays off)

* serializable transaction isolation (something like but not exactly like ssi, rather than 2pl)

* ieee 754-2008 decimal floats

* undo-based mvcc (writers don't block readers)

* group (network) sync replication

* paxos based failover

* lua stored procs

Interesting technology and I'm very happy to see it open-sourced. Kudos to the team. (I used this when I worked there. Few firms can pull off something like this in-house; they could. You wouldn't believe how much data they store in this thing.)

microcolonel 9 years ago | |

You know, I never really understood the point of decimal floats. Why use floats at all if the point is to express beeps and dollar ppm? You might as well just pick a fraction size and use uint64_t to my mind, or just some bigint type (which is still going to be faster than decimal floats).

macdice 9 years ago | | |

The point is that it's hard to pick the fraction size. US national debt vs Italian lira/Swiss franc exhange rate, humans generate numbers of wide ranging scales and yet it's convenient for computers to deal in fixed size datatypes. Hence floating scale. But yeah, it's a compromise. I expect these new types to catch on and be added to various language standards over the coming years, but we'll see.

DenisM 9 years ago | | |

Small parts, like screws, can be priced in microdollars for accounting purposes. At the same time the US GDP is on the order of $20 trillion. 64 bit is not large enough to fit both of these types of numbers.

Makes using SQLITE a real pain...

NickGerleman 9 years ago |

This is really cool but I'm curious why Bloomberg would need this. Ie, what special needs does Bloomberg have that would lead to a primarily non-engineering company investing the resources to create this. Was there nothing off the shelf that would have fit their needs?

I don't mean that in a derogatory way, I'm just curious what motivated making this.

boxfish 9 years ago |

What's the motivation for Bloomberg to open-source this?

rusanu 9 years ago | |

Probably the same motivation Yahoo had to release Hadoop, FB to release Hive, Netflix to release so much of their libs and so on and so forth:

- if nothing else, it does no harm (no 'secret sauce' competitors could benefit from)

- it buys karma (think recruiting goodwill)

If the project catches on though then there are many advantages:

- it can spark a self-sustained ecosystem that can further drive the product, at much lower cost for original creator (think Hadoop leading to Cloudera, Hortonworks etc). Product improves, bugs are fixed, toolset matures

- newhires come with know-how to use your internal tools, lower ramp up, better productivity. Anecdotal, but when I was at Microsoft no newhire knew how to use the internal Cosmos stuff, and even among old timers more folk were familiar with Hadoop...

ihenriksen 9 years ago |

I see Comdb2 requries SQLite to install. So, I'm guessing Comdb2 is a distributed storage engine for SQLite, or?

mponomar 9 years ago | |

There's a heavily modified SQLite embedded in Comdb2 for query parsing/planning. There's some auxiliary tools that can optionally use SQLite (or Comdb2). It'll run without installing an SQLite package.

kanwisher 9 years ago | |

I believe originally it wasn't sql, and they added SQL by using the sqlite parsing engine for SQL. They are a massive contributor to the sqlite project. It is in no way supposed to be compatible with sqlite

czinck 9 years ago | | |

I think you're mixing 2 things. This is a complete rewrite of an old key-value store (hence the 2 in the name) but comdb2 was always SQL. Comdb2 shares a few things with comdb, but they're really just to make migration easier (the preference for tags in csc2 files instead of usual DDL, and then a tag based API that it looks like we got rid of for this release). Under the covers comdb2 is completely different and as far as I know shares no code with comdb.

ihenriksen 9 years ago | | |

So, you can run Comdb without SQLite?

nodesocket 9 years ago |

Somewhat annoying you have to copy data to nodes manually:

    copycomdb2 mptest-1.comdb2.example.com:${HOME}/db/testdb.lrl

This sort of stuff is why I loved RethinkDB. They handled all these complexity details for you.

ketralnis 9 years ago | |

I'm not too offended by letting the administrator figure out the best way to do an uncommon operation. As a point of comparison, Cassandra does have a proper way to bootstrap new nodes but I've found that in many cases it's better to short circuit it and rsync the initial data myself (and use its repair functionality to clean up the mess).

Some reasons include throttling load on the "old" servers, better feedback on progress, the ability to pause/resume, or even being able to do it faster than the DBMS can e.g. by snapshotting the disk on the source machine and making a CoW clone of it. Heck, if you're running your own hardware and feeling a little reckless, pull out one of the drives from the source machine's RAID mirror and you've already got a full clone right there.

I guess you could build all of that into the DBMS, but it's a rather specialised manual operation that's not happening all that often and it's one of the cases that the administrator almost certainly does know better

coredog64 9 years ago | | |

How does that work? As near as I can figure, you need to have all the sstable files from all nodes in a rack on disk. Most will be discarded on "nodetool cleanup", but I would expect it to have to rewrite all the files due to the new token range.

dankohn1 9 years ago | |

RethinkDB is still around (present tense). It has been re-licensed under Apache-2.0 and a community is building to move it forward.

https://www.joyent.com/blog/the-liberation-of-rethinkdb

gleenn 9 years ago |

Nice they have a JDBC driver... makes it a lot easier to hook into. After looking at all the C++ jobs on Bloomberg's website, makes sense the schema format looks C++ish.

pjmlp 9 years ago | |

They have a few people on the ANSI C++ process, including Bjarne. :)

maxlybbert 9 years ago | | |

I think Bjarne works for Morgan Stanley. Bloomberg has John Lakos, Alisdair Merideth, Dietmar Kuhn and others (although I'm not sure which of those are officially on the committee).

shusson 9 years ago |

I wonder how this compares to CockroachDB

brian_herman 9 years ago |

Neat! It supports wsl! Even better! O_o

-- have had NoSQL with stored procs(eg Goldman S, early to mid 90s ), -- in-house grown programming lang (APL+ based in Morgan Stanley) emphasizing vector-based operations -- Smart contracts https://www.lexifi.com/product/technology/contract-description-language (where a contract is represented as algorithm specified in a domain specific language) ). This was way before etherium, started, I believe, in Credit Swiss (but not sure) -- one of the fastest time series database (kdb+) -- Data science and machine learning (modeling risk and valuations)