I feel we hold up single-observability-solution as the Holy Grail, and I can see the argument for it- one place to understand the health of your services.

But I've also been in terrible vendor lock-in situations, being bent over the barrel because switching to a better solution is so damn expensive.

At least now with OTel you have an open standard that allows you to switch easier, but even then I'd rather have 2 solutions that meet my exact observability requirements than a single solution that does everything OKish.

Depending on your usage it can be prohibitively expensive to use datadog for everything like that. We have it for just our prod env because it's just not worth what it brings to the table to put all of our logs into it.

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time.

It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to do logs (and metrics sometimes) in cloudwatch but spending 95% of my time in honeycomb - I'd pick honeycomb every time

Eh, personally I view honeycomb and datadog as different enough offerings that I can see why you'd choose to have both.

> It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability.

One of the biggest features of AWS which is very easy to take for granted and go unnoticed is Amazon CloudWatch. It supports metrics, logging, alarms, metrics from alarms, alarms from alarms, querying historical logs, trigger actions, etc etc etc. and it covers each and every single service provided by AWS including metaservices like AWS Config and Cloudtrail.

And you barely notice it. It's just there, and you can see everything.

> One of the terrible mistakes I see companies make with this tooling is fragmenting like this.

So much this. It's not fun at all to have to go through logs and metrics on any application,and much less so if for some reason their maintainers scattered their metrics emission to the four winds. However, with AWS all roads lead to Cloudwatch, and everything is so much better.

We tried this about a year and a half ago and ended up going somewhat backwards into DD entrenchment, because they've decided that anything not an official DD metric (that is, collected by their agent typically) is custom and then becomes substantially more expensive. We wanted a nice migration path from any vendor to any other vendor but they have a fairly effective strategy for making gradual migrations more expensive for heavy telemetry users. At least our instrumentation these days is otel, but it's the metrics we expected to just scrape from prometheus that we had to dial back and start using more official DD agent metrics and configs to get, lest our bill balloon by 10x. It's a frustrating place to be. Especially since it's still not remotely cheap, just that it could be way worse.

I know this isn't a DataDog post, and I'm a bit off topic, but I try to do my best to warn against DD these days.

Author here, thanks for the question! The current split developed from the personal preferences of the engineers who initially set up our observability systems, based on what they had used (and liked) at previous jobs.

We're definitely open to doing more consolidation in the future, especially if we can save money by doing that, but from a usability standpoint we've been pretty happy with Honeycomb for traces and Datadog for everything else so far. And, that seems to be aligned with what each vendor is best at at the moment.

Yeah, the ability to send to multiple sources is quite powerful and most of this comes from the configurability of Otel Collector [1].

If you are looking for a open source backend for OpenTelemetry, then you can explore SigNoz[2] (I am one of the founders) We have a quite a decent product for APM/tracing leveraging opentelemerty native data format and semantic convention.

[1]https://opentelemetry.io/docs/collector/ [2]https://github.com/SigNoz/signoz

> mimir because of scale & self-host options

Have you looked at VictoriaMetrics [0] before opting for Mimir?

[0] https://victoriametrics.com/blog/mimir-benchmark/

you can submit opentelemetry traces to datadog which should be the equivalent of apm/npm, though maybe with a less polished integration.

Thanks! For logs, we actually use github.com/segmentio/events and just implemented a handler for that library that batches logs and periodically flushes them out to our collector using the underlying protocol buffer interface. We plan on migrating to log/slog soon, and once we do that we'll adapt our handler and can share the code.

First you had logs. Everyone uses logs because it's easy. Logs are great, but suddenly you're spending a crapton of time or money maintaining terabytes or petabytes of storage and ingest of logs. And even worse, in some cases for these logs, you don't actually care about 99% of the log line and simply want a single number, such as CPU utilization or the value of the shopping cart or latency.

So, someone says, "let's make something smaller and more portable than logs. We need to track numerical data over time more easily, so that we can see pretty charts of when these values are outside of where they should be." This ends up being metrics and a time-series database (TSDB), built to handle not arbitrary lines of text but instead meant to parse out metadata and append numerical data to existing time-series based on that metadata.

Between metrics and logs, you end up with a good idea of what's going on with your infrastructure, but logs are still too verbose to understand what's happening with your applications past a certain point. If you have an application crashing repeatedly, or if you've got applications running slowly, metrics and logs can't really help you there. So companies built out Application Performance Monitoring, meant to tap directly into the processes running on the box and spit out all sorts of interesting runtime metrics and events about not just the applications, but the specific methods and calls those applications are utilizing within their stack/code.

Initially, this works great if you're running these APM tools on a single box within monolithic stacks, but as the world moved toward Cloud Service Providers and containerized/ephemeral infrastructure, APM stopped being as effective. When a transaction starts to go through multiple machines and microservices, APM deployed on those boxes individually can't give you the context of how these disparate calls relate to a holistic transaction.

So someone says, "hey, what if we include transaction IDs in these service calls, so that we can post-hoc stitch together these individual transaction lines into a whole transaction, end-to-end?" Which is how you end up with the concept of spans and traces, taking what worked well with Application Performance Monitoring and generalizing that out into the modern microservices architectures that are more common today.

Author here. This decision was more about ease of implementation than anything else. Our internal application logs were already being scooped up by GCP because we run our services in GKE, and we already had a GCP->Datadog log syncer [1] for some other GCP infra logs, so re-using the GCP-based pipeline was the easiest way to handle our application logs once we removed the Datadog agent.

In the future, we'll probably switch these logs to also go through our collector, and it shouldn't be super hard (because we already implemented a golang OTel log handler for the external case), but we just haven't gotten around to it yet.

[1] https://docs.datadoghq.com/integrations/google_cloud_platfor...

Their collector is used to send infrastructure logs to GCP (instead of Datadog).

My guess is this is to save on costs. GCP logging is probably cheaper than Datadog, and infrastructure logs may not be needed as frequently as application logs.

Have you checked out Jaeger [1]? It is lightweight enough for a personal project, open source, and featureful enough to really help "turn on the lightbulb" with other engineers to show them the difference between logging/monitoring and tracing.

[1] https://www.jaegertracing.io/

grafana cloud, honeycomb, etc have free tiers, though you'll have to watch how much data you send them. or you can self host something like signoz or the elastic stack. frontend will typically go to an instance of opentelemetry collector to filter/convert to the protocol for the storage backend.

I've never heard the term "pre-observability", what does that mean?

Observability is about logs and metrics, and pre-observability (I guess you mean the high-level-only records simpler environments keep) is also about logs and metrics.

Anything you register to keep track of your environment has the form of either logs or metrics. The difference is about the contents of such logs and metrics.

Thanks, that's nice to know VM can accommodate that. A migration will be something we will have consider if Mimir and OpenTelemetry force us to use joins for all our queries.

They maybe trying to address label cardinality but their approach seems like they are throwing the baby out with the bath water. The developer experience suffers as a result because from a dev pov, resource attributes are added to the metric yet this relationship isn't transferred when translated to Prometheus metrics.

We've migrated away from the prometheus exporter to the prometheus remote write exporter as I'd like a completely "push" based architecture. Ideally I would have liked to be completely otlp but can't for the reasons already explained. So I use promethus remote write instead in to Mimir.

The transform processor could be useful if they ever deprecate the resource_to_telemetry_conversion flag, but its still a pain point because it hinders a developers autonomy, and requires a whitelist of labels to be maintained on the collectors by another team.

Thanks for the detailed response.

I am surprised there is no gauge update API yet (instead of callback only), this is a common use case and I don't think folks should be expected to implement their own. Especially since it will lead to potentially allocation heavy bespoke implementations, depending on use case given mutex+callback+other structures that likely need to be heap allocated (vs a simple int64 wrapper with atomic update/load APIs).

Also I would just say that the fact the APIs differ a lot to more common popular Prometheus client libraries does beg the question of do we need more complicated APIs that folks have a harder time using. Now is the time to modernize these before everyone is instrumented with some generation of a client library that would need to change/evolve. The whole idea of an OTel SDK is instrument once and then avoid needing to re-instrument again when making changes to your observability pipeline and where it's pointed. This becomes a hard sell if OTel SDK needs to shift fairly significantly to support more popular & common use cases with more typical APIs and by doing so leaves a whole bunch of OTel instrumented code that needs to be modernized to a different looking API.

the official SDKs will only support an api once there's a spec that allows it.

for const attributes, generally these should be defined at the resource / provider level: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric#WithR...