Hey hn! I'm one of the co-founders of highlight.io, an open source monitoring tool.

Today we're sharing a ClickHouse-based logging solution we've been working on. We wanted to showcase how we built it and share how you could try it out to give feedback. Since we started working on highlight.io, we've been hyper-focused on "cohesion", or ensuring that when you install your monitoring stack, all of the resources in that stack (user interactions, requests, traces, logs, etc.) are connected in a consumable way. We've written up more about our philosophy on this here [1].

We started building towards this by connecting your client-side app and your server-side exceptions with session replay and exception monitoring; i.e. if an error happened in a server-side app, we would make it easy (with session replay) to trace all the steps that a user took leading up to it.

Especially for larger companies using highlight.io, the request to tie in logs came up repeatedly, and we wanted to build this with the same philosophy in mind. Now, you'll see client-side and server-side logs all in one place, brought together in the context of a user session, as well as logs in the context of an error.

Like the rest of our stack, this project is written in Go and Typescript, and for log ingestion/querying, we're using ClickHouse [2]. Before deciding on ClickHouse, we were planning to use OpenSearch (an aws fork of elasticsearch [3]) for this part of our product, but as our traffic has increased, we encountered quite a few pains with write throughput for OpenSearch. After evaluating a few options, we eventually landed with ClickHouse (their cloud offering was icing on the cake), which has also proven to be much more cost-effective so far.

Building with ClickHouse from scratch has been an exciting journey. Eric (the mastermind behind this project) wrote a blog post [4] on a handful of ClickHouse learnings we've gathered since starting the project.

For those wanting to try out the product locally, you can run the following commands [5]:

git clone --recurse-submodules https://github.com/highlight/highlight cd highlight/docker; ./run-hobby.sh;

To send logs to highlight, you can use your own OpenTelemetry implementation [6] or use our SDKs [7] which provide lightweight wrappers over OTEL.

Like the rest of highlight.io, we plan to make money from this with our hosted cloud offering. For those interested in trying out the cloud-hosted version, you can get setup at app.highlight.io.

To open the floor for feedback, we would love to get some thoughts on what we've built so far. Beyond that, what are parts of a logging product you wish you had with your current setup? And are there any notable pain-points of using a hosted monitoring product? (We're toying with the idea of an enterprise deployment). Excited to hear from everyone.

[1]: https://highlight.io/docs/general/company/product-philosphy

[2]: https://clickhouse.com

[3]: https://news.ycombinator.com/item?id=26780848

[4]: https://www.highlight.io/blog/how-we-built-logging-with-clic...

[5]: https://www.highlight.io/docs/getting-started/self-host/self...

[6]: https://www.highlight.io/docs/getting-started/backend-loggin...

[7]: https://www.highlight.io/docs/getting-started/overview#for-y...

user:	liketochill
created:	June 30, 2021
karma:	455