Evolving clock sync for distributed databases

Evolving clock sync for distributed databases(blog.yugabyte.com)

78 points by def- 4 years ago | 17 comments

rdtsc 4 years ago |

> YugabyteDB is the first—and only—100% open-source, hybrid, multi-cloud, distributed SQL database on the planet.

It might be worth pointing out that HCL are patented by Cloudera in 2014 (with a provisional patent application filed in 2013) https://www.freepatentsonline.com/y2015/0156262.html

So while it is 100% open source, depending on your jurisdiction, you could be sued by Cloudera for patent infringement.

It is interesting, also how the referenced paper https://cse.buffalo.edu/tech-reports/2014-04.pdf is from 2014. Wonder if they knew about it each other? It doesn't seem like the researchers from the paper are mentioned in the patent at all.

jeffbee 4 years ago |

I find these articles (and there are dozens of them) somewhat frustrating because they fundamentally misrepresent the role of time sync in distributed databases like Spanner. Spanner doesn't get correctness from tight synchronization, it gets low latency from that. Spanner's correctness comes from estimating the clock error of cluster participants and evicting them when necessary. You can have the latter property without the former, and having the former property without the latter is useless.

rkarthik007 4 years ago | |

Spanner gets both correctness and low latency from tight synchronization. They do COMMIT_WAIT, meaning wait for the max clock skew to pass. But the max clock skew without TrueTime will be around 500ms (which it impractical to wait out). So, the 7ms is lower latency, but 500ms is impractical (more so than just calling it higher latency). And any other technique to drop latency (without HLC) will violate correctness.

Disclosure: co-founder/CTO of YugabyteDB project

dastbe 4 years ago | | |

fwiw i think you’re both saying something very similar. true time has to be correct about max skew in order for it to not break the assumption spanner is built upon. you could also use a looser time bound and still have correctness, but end up with a database that is useless to most/all customers

anonymousDan 4 years ago | | |

Interesting. How does this differ from HLC in practice - in the article you say you use a 500ms max skew for HLC?

vlovich123 4 years ago | | |

What do you think of RIFL/TAPIR?

dbarale108 4 years ago |

Yugabyte ensures transactions consistency for a ACID model in a linear clock synchronization between nodes. Great article!

traceroute66 4 years ago |

Oh joys, the sound of re-inventing the wheel. The old "not invented here" syndrome.

Per the blog, they state they want max 7ms difference between nodes.

I'm no timesync guru, but AFAIK can do that off-the-shelf today using chrony with hardware timestamping or PTP. No need to invent your own.

And if you still insist on doing something different, why not work on improving the existing work being done under the the Open Compute Time Appliance Project (https://www.opencompute.org/projects/time-appliances-project...).

rkarthik007 4 years ago | |

> but AFAIK can do that off-the-shelf today using chrony with hardware timestamping or PTP. No need to invent your own.

Actually, the issue is about the max clock skew guarantees (as opposed to the average or median). Even a single violation of this breaks the ACID semantics. So, we do use chrony, but need all this to ensure there is a max guarantee. We would totally have adopted an existing solution - we did look at all alternatives available.

Disclosure: one of the founders of the YugabyteDB project

traceroute66 4 years ago | | |

How would you say your solution compares to the CockroachDB solution[1] ?

[1] https://www.cockroachlabs.com/docs/stable/architecture/trans...

jeffbee 4 years ago | |

Good luck using PTP in EC2.

yangosoft 4 years ago | | |

Aren't they providing NIC with hardware timestamp? That's bad.