Show HN: Open-source APM with support for tracing, metrics, and logs

Show HN: Open-source APM with support for tracing, metrics, and logs(app.uptrace.dev)

112 points by vmihailenco 3 years ago | 46 comments

Uptrace is an all-in-one tool that supports distributed tracing, metrics, and logs. It uses OpenTelelemetry observability framework to collect data and ClickHouse database to store it.

You can ingest data using OpenTelemetry Protocol (OTLP), Vector Logs, and Zipkin API. You can also use OpenTelemetry Collector to collect Prometheus metrics or receive data from Jaeger, X Ray, Apache, PostgreSQL, MySQL and many more.

The latest Uptrace release introduces support for OpenTelemetry Metrics which includes:

- User interface to build table-based and grid-based dashboards.

- Pre-built dashboard templates for Golang, Redis, PostgreSQL, MySQL, and host metrics.

- Metrics monitoring aka alerting rules inspired by Prometheus.

- Notifications via email/Slack/PagerDuty using AlertManager integration.

There are 2 quick ways to try Uptrace:

- Using the Docker container - https://github.com/uptrace/uptrace/tree/master/example/docke...

- Using the public demo - https://app.uptrace.dev/play

I will be happy to answer your questions in the comments.

derN3rd 3 years ago |

Nice to see so many new projects in the area of APM in the last few months.

We recently tried Signoz and Grafana Tempo and while I can't say something about uptrace yet (will definitely try it out) I want to list some pros and cons about them.

Grafana Tempo

Pros:

- Easy and smooth integration into our existing Grafana instance, no additional frontend needed

- No new storage engine needed (No additional Clickhouse, Postgres, etc) as it saves its data to S3

- Supports OTLP

Cons:

- Search is limited by param size and unique params (as its baked to be indexed)

- Ingestion is not in real time, but configurable (time to finish span)

Signoz:

Pros:

- Supports OTLP

- Integrates Logs and Metrics within the same service (for Grafana you need Loki then)

- Supports real time querying

Cons:

- Uses new storage engines (or extends the software stack) with adding ClickHouse

- Adds an additional frontend (might not be relevant for everyone)

- Doesn't provide SSO yet, so you need to manage users differently

Interesting to see, that UpTrace also chose ClickHouse (btw I love ClickHouse!)

Some questions:

- Can I easily disable certain features? (e.g. alerting)

- Is there support for SSO for self-hosted installation?

- Are there any recommendations for scaling (e.g. benchmarks) on how many spans/s are supported on what hardware?

Thanks in advance!

vmihailenco 3 years ago | |

Thanks for the feedback!

>- Are there any recommendations for scaling (e.g. benchmarks) on how many spans/s are supported on what hardware?

With Uptrace, I was able to achieve 10k spans / second on a single core by running the binary with GOMAXPROCS=1. That is 1-3 terrabytes of compressed data each month which is more than most users need.

Practically, you are limited by the $$$ you are willing to spend on ClickHouse servers, not by Uptrace ingestion speed.

So my recommendation is to scale Uptrace vertically by throwing more cores at it. That will allow you to go very very far.

>- Is there support for SSO for self-hosted installation?

So far the only way to add news users is via the YAML config. We are considering to add a REST API or a CLI tool for the same purpose, but it is not clear how that would work with the YAML.

Regarding the SSO, it would be nice if you can provide an app that already does that so we can better estimate the complexity. But so far we don't have such plans.

>- Can I easily disable certain features? (e.g. alerting)

Yes, most YAML sections just can be removed / commented out to disable the feature.

derN3rd 3 years ago | | |

Thanks for the answer.

I have no other good examples for SSO than Grafana.

But something I would love to see more for logins are Application Tokens. We use Cloudflare Access for Team related logins, which will send such a token in the header, so the application can use it to authorize a user and by which group a user is in it can enable/disable features

https://developers.cloudflare.com/cloudflare-one/identity/au...

This solved our needs for multiple sign in options, as all is now managed through Cloudflare access, but this is obviously not a solution for everyone

pranay01 3 years ago | |

thanks for the mention. I am one of the maintainers at SigNoz [1].

Thanks for laying out the points in Pro section. We also recently launched logs witg v0.11.0 so you may want to give it a try again - we now have have metrics, logs and traces in a single app.

Would love to understand a few points in more details you have mentioned in Cons for SigNoz

> - Uses new storage engines (or extends the software stack) with adding ClickHouse Can you explain a bit more on the concern here?

> - Doesn't provide SSO yet, so you need to manage users differently

This is in our roadmap and we will be shipping it soon.

[1] https://github.com/SigNoz/signoz

Havoc 3 years ago | |

Didn’t know about the grafana option thanks. Interesting - so one can do metrics dash, logs and tracing in one interface!

vmihailenco 3 years ago |

The latest Uptrace release introduces support for OpenTelemetry Metrics which includes:

- User interface to build table-based and grid-based dashboards.

- Pre-built dashboard templates for Golang, Redis, PostgreSQL, MySQL, and host metrics.

- Metrics monitoring aka alerting rules inspired by Prometheus.

- Notifications via email/Slack/PagerDuty using AlertManager integration.

There are 2 quick ways to try Uptrace:

- Using the Docker container - https://github.com/uptrace/uptrace/tree/master/example/docke...

- Using the public demo - https://app.uptrace.dev/play

I will be happy to answer your questions in the comments.

0JzW 3 years ago | |

this looks amazing! i would definitely like to use this for log monitoring. however, i have a question. is it possible to get logs for individual docker containers?

vmihailenco 3 years ago | | |

It is possible using Vector Logs which Uptrace supports out-of-the-box, for example:

- https://vector.dev/docs/reference/configuration/sources/dock...

- https://uptrace.dev/get/ingest/vector.html

If you are having troubles making it work, feel free to open an issue on Github and I will provide a complete example.

bovermyer 3 years ago |

I see lots of new tracing options these days, and that seems to have taken over the "APM" term.

I still have yet to see new profiling options. When I think of APM, I think of CPU profiling and automatic instrumentation of black box systems, not request tracing. I should be able to see which function calls are slow/problematic, without having to add code to the application.

vmihailenco 3 years ago | |

I use memory profiling with Go and it is indeed very useful. I think that whatever Go already provides is enough, but I guess Uptrace could try to automate some things and/or provide some fancy UI.

But I find CPU profiling a lot less useful, because production profiles tend to be too broad and it is hard to make sense from them, for example, Uptrace profile mostly consists of memory allocations and network calls.

So I would not say that CPU profiling is superior/better/can replace tracing.

bovermyer 3 years ago | | |

Tracing is a piece of the puzzle and necessary. Profiling occupies a different part of the monitoring ecosystem.

nijave 3 years ago | |

I think there is some interesting work being done with eBPF in the profiling space

jonasdevops 3 years ago |

Before you try, please make sure you are comfortable with their license - https://github.com/uptrace/uptrace/blob/master/LICENSE (Business Source License 1.1), which as License says "The Business Source License (this document, or the “License”) is not an Open Source license"

KronisLV 3 years ago |

This seems like a pretty cool project!

Currently using Apache Skywalking myself, because it's reasonably simple to get up and running, as well as integrate with some of the more popular stacks: https://skywalking.apache.org/

I do wonder how ClickHouse (which Uptrace uses) would compare with something like ElasticSearch (which is used by Skywalking and some others) and how badly/well an attempt to use something like MariaDB/MySQL/PostgreSQL for a similar workload would actually go.

I mean, something like Matomo Analytics already uses a traditional RDBMS for storing its data, albeit it might be an order of magnitude or two off from the typical APM solution.

vmihailenco 3 years ago | |

When compared with ElasticSearch, ClickHouse can handle the same amount data using 10x less resources and that is not an exaggeration. It is even worse with MariaDB/MySQL/PostgreSQL.

I guess ElasticSearch is still relevant when it comes to searching text, but ClickHouse is much faster when it comes to filtering and analyzing the data.

Give ClickHouse a try and you won't be disappointed.

https://benchmark.clickhouse.com/

tylergetsay 3 years ago |

I think the log interface should be optimized for keyboard navigation and larger screens. On my 4k monitor it only takes up 1/2 the width and only shows 10 lines at a time, id expect closer to ~100

vmihailenco 3 years ago | |

Thanks for the feedback. Any projects that you could recommend that do it right?

tmd83 3 years ago |

I wonder if anyone can answer some question on distributed tracing for me.

The difference between old days of APM vs. tracing as I understand is two things.

1. Originally APM was single process and it was language aware, usually do sampling stacktrace to find where times are being taken and some very well know place to instrument for exact timing say response time or query time.

Tracers are more working by instrumenting methods of framework/servers/runtime at well known point and getting the timing. In man ways it's a lot more coarse as it might know of a hot loop that I have in my code. But it can trace very well with exact timing at framework boundary like web, cache, db etc.

2. The APM were primarily single process and couldn't really show a different service/process which doesn't work in a micro-service/distributed world.

The way I understand it is that Tracers would allow me to narrow down to the service/component very easily. Whether I can find out why that component is slow might not be as easy (not sure what granularity tracing happens inside a component).

I wonder if this understanding of mine is correct.

The second thing I am really unsure about is sampling and overhead. What's the usual overhead of a single tracing (I know it's variable) but generally are they more expensive at a single request level. Also do they usually sample and is there a good/recommend way to sample this. I forgot exactly who but (probably NewRelic) was saying they collect all traces (like every request?) and discard if they are not anomalous (to save on storage). But does that mean taking a trace is very cheap? And is that end of the request sampling decision something that's common or that's a totally unique capability some have.

PeterZaitsev 3 years ago |

False Advertising!

BSL Licensed is not Open Source. To Be fair Utrace restrictions are relatively light but it is still Source Available project not Open Source

nik736 3 years ago |

Nice! Exactly what I've been looking for, will give it a try for sure. Sentry eats a lot of resources so I was looking for an alternative.

vmihailenco 3 years ago | |

Thanks! Don't hesitate to send any feedback you have so we have a chance to improve :)

edf13 3 years ago |

Looks nice... I'm a bit out of touch in this space but my last solution for similar would be Datadog. How does this compare?

vmihailenco 3 years ago | |

DataDog has a high learning curve and can be rather expensive if you need to monitor a lot of hosts and microservices.

Uptrace tries hard to stay simple while providing almost the same set of features. It can also be self-hosted without paying anything which can save a lot of money.

Uptrace aims to be an open source alternative to DataDog, but realistically we are not there yet.

xyzzy_plugh 3 years ago |

I've been out of the loop for a while but...

> OpenTelemetry Protocol (OTLP)

> OTLP

> OLTP

I'm going back to bed.

xfer 3 years ago |

Anyways to export dashboard for public viewing, maybe even static image? It looks like all drawing is done client side at present.

vmihailenco 3 years ago | |

Echarts which we use supports exporting charts as images so it probably can be added relatively easy. Embedding is another possible option.

ram_rar 3 years ago |

can you elaborate more on why clickhouse for backend? And what challenges if any are you facing with clickhouse?

wdb 3 years ago |

How does it compare to Opstrace? (www.opstrace.com)