The config makes this easy, but my favorite part is the fact that the CPU and MEM of the vector processes barely even registers on my metrics charts. I can't even tell you what their actual and requested resources are because I haven't bothered to look in a while.
It's one thing I never have to worry about. I could use more of those.
I think the interesting part is the overlap between observability metrics for operations and expanding in to BI metrics with the same tools.
I built Logflare so you could get your structured logs into BigQuery directly, so now you this :)
Hopefully we'll have a Vector sink soon, but until then I think they support POSTing batches to HTTP endpoints so you can do that and we'll take any JSON and go straight to BQ with it after migrating your schema for you (automatically based on the incoming payload shape).
Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.
I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value this kind of feedback and try to address it as promptly as possible.
The diagram and succinct summary give you a very high level view, but the Docs link at the top takes you this:
I get that you want to sell this to executives, but the people that point executives to products to buy are the developers like me, and if don't understand what it does, I can't see executives finding your product.
Just give examples of what on-the-ground problems is solves, converts kafka messages to s3 objects or what?
Lack of novel data enrichment is not a problem that I have, but I could do with something that streams k8s pod logs to kafka or s3 or whatever. Does this solve that?
That being said, we do know of a few cases where Vector's in-flight processing and routing capabilities were enough that a full Kafka-based pipeline was no longer needed. This ability to push computation out to the edge and reduce the need for centralized infrastructure is one of the aspects of Vector that we're most excited about.
The "collect, transform, and route all your logs, metrics, and traces" bit is our most succinct explanation of what Vector does, but I'll admit it's still not as clear as we'd like. To expand it slightly, Vector is a tool to collect observability data (logs, metrics, etc) from wherever it's generated, optionally process that data in-flight, and then forward it to whatever upstream system you'd like to consume it. It does this by providing a variety of different components that you configure into whatever pipeline you need. In your example, you could use our new k8s source and plug it into our Kafka sink, our S3 sink, or both.