Building a Scalable Postgres Metrics Backend Using the Citus Extension

Building a Scalable Postgres Metrics Backend Using the Citus Extension(citusdata.com)

96 points by mmcgrana 9 years ago | 10 comments

devbug 9 years ago |

If you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus.

https://www.youtube.com/watch?v=NVl9_6J1G60

doh 9 years ago |

We[0] just recently moved from Hbase to Postgres w/ Citus. Still not fully settled in, but got incredible support from the Citus team (seriously amazing guys) and things are looking much brighter than before.

On a 8 worker nodes (16 cores, 80GB ram, 3TB SSD), we're at around 20k inserts/second, while most of the selects are running at around 100ms (besides some counts and complex queries).

If you have a lot of data and know Postgres, give it a chance.

[0] https://pexe.so

koolba 9 years ago | |

> On a 8 worker nodes (16 cores, 80GB ram, 3TB SSD), we're at around 20k inserts/second, while most of the selects are running at around 100ms (besides some counts and complex queries).

Is that 20K separate transactions or are some of the 20K inserts grouped together?

doh 9 years ago | | |

Separate (going through the master).

Jake232 9 years ago | |

This is a super cool product. Good job.

locusm 9 years ago | |

Interesting product - good luck :)

sciurus 9 years ago |

If you're looking for a metrics backend built on traditional Postgres, check out Tgres.

http://grisha.org/blog/2016/07/29/state-of-tgres-2016/

GrayTShirt 9 years ago | |

+1 for tgres https://github.com/tgres/tgres On an aside, I found his posts on holt-winters very informative.

kornish 9 years ago |

For anyone with questions about how to properly shard a table, how to fit a problem to a distribution strategy, or general questions about deployment, Citus has a Slack channel which is active and extremely welcoming. You can request an invite here: https://slack.citusdata.com/

Disclaimer: don't work there, but on the Slack and the Citus team is nothing short of awesome.

dataloopio 9 years ago |

I had a look at Citus when evaluating time series databases and the documents said that work was still under way on masterless setup for faster ingestion (up to 500k metrics /sec).

https://docs.citusdata.com/en/v5.2/performance/scaling_data_...

If that has changed I'd add it to my table.

https://docs.google.com/spreadsheets/d/1sMQe9oOKhMhIVw9WmuCE...