Real World Database Latency

42 points by evanweaver 4 years ago | 8 comments

aeyes 4 years ago |

What's going on with the latency for the US region on the status page? https://status.fauna.com/#day

1 second for writes and 400ms for reads? Looks like this has been going on for a week but there is nothing posted about it on the status page.

tysont 4 years ago | |

Nice catch! I investigated this today with the team and we realized that metrics from a dev environment (which isn't representative of our production topology) were being included in our status page internal latency graphs for the US/EU Region Groups. We addressed the issue - you can see the clean data on status.fauna.com.

evanweaver 4 years ago |

"There are three kinds of lies: lies, damned lies, and benchmarks." - Mark Twain

What other methods does the community use for measuring distributed latency?

PeterCorless 4 years ago | |

"We investigate the issue of coordinate stability over time and show that coordinates drift away from their initial values with time, so that 25% of node coordinates become inaccurate by more than 33 ms after one week. However, daily re-computations make 75% of the coordinates stay within 6 ms of their initial values."

That's the intro from a 2007 paper from Google:

https://static.googleusercontent.com/media/research.google.c...

igouy 4 years ago | |

After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, 'Lies--damned lies--and statistics,' still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of.

Leonard Henry Courtney, 1895

tysont 4 years ago |

To add a bit of color from the author, the content in the article was informed by my experience building distributed systems at Microsoft, Amazon, and Riot Games. At Riot we sharded our playerbase for a number of reasons (including complexity of matchmaking), but we may have chosen a different approach if we had a service like Fauna that could push data closer to players at the edge. With centralized relational databases, cross-region calls to load account info, entitlements, etc. would have been a very jarring experience in the client.

I've seen a number of database companies publish blog posts and put out tweets offering a competing point of view: that single-region is enough and multi-region doesn't matter. Would love to engage with that crowd here and get their perspective on the content in the blog post.

omeze 4 years ago | |

Just clarifying - does this mean matchmaking for League had geographic affinity (for Valorant this is true I believe)? And was that driven by a desire to ensure tight matchmaking performance latency guarantees or in-game latency goals?

tysont 4 years ago | | |

League sharded players to geographic regions and didn't allow matches between players in different regions. Geographic location wasn't a consideration in matchmaking otherwise AFAIK. I left Riot when Valorant was still under development and can't speak to how matchmaking works, unfortunately.