My experience of Prometheus is outside of containerised environments and I enjoyed Prometheus to begin with.

Then it starts to grow and everything becomes increasingly more difficult until your patience runs out, or you just can't get it to work any more, and you split it in two, now you have two headaches but at least you can get it to work.

Your now 12 months maybe 2 years in and you realise two things, that stupidly expensive Dynatrace install works out cheaper than all the engineering time you've thrown at rolling your own, the opportunity cost makes it a no brainer, and you'd pay out of your own pocket to be doing something more interesting...

I used think building everything yourself was the way to go, it is more fun (at least until you hit operations). Now I'm older, maybe wiser, if it's not core business, I ship it off to a vendor and let them deal with it.