This looks great. My primary attraction is possibly low memory footprint of this program over Filebeat. Secondary attraction is how easy it appears to enable transformations.
Now, if I can make a suggestion for your next/additional project..... A neat system metric collector in Rust that can export to Prometheus with same principles.
Low memory footprint,
Rust,
Single binary,
Customizable with a single config file without spending hours in manuals,
Stdin, Stderr -> transform -> Prometheus.
I’m learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one.
Cheers on this open source project. I will contribute whatever I can. Thanks!!
It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!
There was discussion earlier this year about creating a design doc for OpenCensus to handle logs. I'm not sure if that got finished, or it was sidelined while the OpenCensus & OpenTracing merger was worked on. Both projects will combine under the OpenTelemetry name.
I've been quite happy with the OpenCensus instrumentation SDKs.
I think the Logs & Metrics space is interesting, especially because there is so much overlap, both are just ways of representing data about an event that occurred in the software.
OpenCensus is fairly widely backed: Google, Microsoft, Etsy, Scalyr...[1]
[0] https://github.com/census-instrumentation/opencensus-service...
A big part of the reason we started building Vector was to integrate that kind of functionality into a larger project, so people wouldn't have to get over the hump of discovering, setting up, and rolling out a whole separate tool.
We're definitely not as mature as mtail yet, but we're working really hard to get there.
Veneur is more metrics-focused, but might offer inspiration as you work on metrics support in Vector - in particular the SSF source, internal aggregation, and Datadog and SignalFX sinks.
As you mentioned, it seems they've focused more on metrics out of the gate, while we've spent more of our time on the logging side of things (for now). We're working to catch up on metrics functionality, but interoperability via SSF is an interesting idea!
I did really like Kibana, ultimately, we had to ditch it (because of ditching ES). Of course, this was a good thing, as I more than once degraded ingest the ES cluster by just using Kibana to do some aggressive filtering. Clickhouse handles these without problem.
I think a more complete world view may be to pipe logs into kafka, and ingest them into Clickhouse/Druid for different types of analysis/rollups.
Our current logging volume exceeds ~10b log lines per day now. Clickhouse handles this ingest almost too well (we have 3 16 core nodes that sit at 5% CPU). This is down from a... 20ish node ES cluster that basically set pegged on CPU... and our log volume then was ~1b/day.
For more ad-hoc, we just use the clickhouse-cli to query the dataset directly. We are tangentially investigating using superset with it.
Specifically, I'm ingesting nginx logs in JSON format, cleaning up invalid UTF8 bytes (usually sent in the forwarded-for header to exploit security vulnerabilities), and sending to elasticsearch on an automated 90 day retention policy (daily indexes).
Seems like a fairly common use case for webservers.
I don't know why author didn't put correctness tick mark on it.
https://github.com/timberio/vector-test-harness/tree/master/...
Definitely open to feedback on what we're doing wrong.
https://github.com/influxdata/telegraf
Biggest thing that pops out to me is LUA engine (seems amazing :) )
But to answer your question, telegraf is very heavily metrics focused, and their logging support appears to be limited (reducing logs to metrics only). Vector is _currently_ focused on logging with an eye towards metrics, but still has work to do on the metrics front.
For example, we opened the door with the `log_to_metric` transform (https://docs.vector.dev/usage/configuration/transforms/log_t...) to ensure our data model supports metrics, but we still have a lot of work to do when it comes to metrics as a whole. Our end goal is to eventually replace telegraf and be a single, open, vendor neutral solution for both logs and metrics.
Happy to clarify further :)
Does either of Filebeat or Logstash support config hot reload, as mentioned in the Vector's doc? https://docs.vector.dev/usage/administration/reloading
Edit - Found It - https://www.elastic.co/guide/en/logstash/current/reloading-c...
https://github.com/timberio/vector-test-harness/tree/master/...
Logstash is not graceful. Our testing shows that they basically shut it down and start it again.
If yes, how comes log processing runs at only so low throughput in general?
That is not to talk down your achievements (as per your benchmark page, you do better than similar projects in terms of throughput), but I'm genuinely curious why modern machines that have 40 Gbit/s memory bandwidth are capped at (in your case) 76.7Mbit/s. What's the bottleneck?
Given that the reported values don't care about the "m" (which means "milli" -- clearly doesn't make sense for bytes), I don't think we can rely on the casing of the "b" to tell us the answer.
There are obviously plenty of ways to accomplish that same thing today, but we believe Vector is somewhat unique in allowing you to do it with one tool, without touching your application code or nginx config, and with enough performance to handle serious workloads. And Vector is far from done! There's a ton more we're working to add moving forward (thinking about observability data from an ETL and stream processing perspective should give you a rough idea).
So is Vector like Splunk forwarder or more than that?
https://github.com/mozilla-services/hindsight
Unfortunately deployment of Hindsight isn't as nice as Heka since you need to compile it yourself with all the Lua extensions you need, and the documentation is very disorganized.
Vector looks great on those counts, will be excited to try it if they get features like reliable Vector-Vector transport and more flexible file delimiters.
(Disclosure: I work for Mozilla on the team that runs services used by firefox users and developers)
While better performance is always great, most are already plenty fast for the majority of use cases.
The main power comes from the multitude of inputs and outputs. Vector has a lot of catching up to do there. But if they manage to offer a noteworthy performance gain... one more is always a good thing.
PS: the Logstash numbers seem suspiciously low. I'd bet it's some JVM config issue. Logstash can come to a crawl if it does not have enough memory.
As for the multitudes of inputs/outputs, covering the 95% most-used sources and sinks is a great starting point. I think Vector got that list right in this case.
Regardless, Vector looks very nice and I'll be testing it out :)
Reported an issue in their test harness.
Also, the documentation seems to miss information about which sinks support TLS?
We're currently looking for a distributed-over-the-internet logging setup and are interested in evaluating Rsyslog/RELP alternatives.
Do you have specific module for it?
Vector's perspective is that your applications and infrastructure are already emitting all kinds of interesting data via logs, metrics, etc, and the primary challenge is to collect, enrich, and manage the storage of that data. We have no plans to integrate Vector into your application or introduce some kind of Vector-specific method of exporting data.
We'll definitely be watching OpenTelemetry as it moves forward and would very much like to be a compatible part of that ecosystem. To the degree that they use common open standards for their communication protocols, that could just fall out naturally.
I was mentally debating between trying to find a schema for our logs, and store them in a database where it can be queried efficiently from
Vs
Throwing logs into ELasticSearch in a lazy way and let it index the whole thing to enable us do full-text search on logs. But, with a limitation of only have a few days worth of data in ES indexes.
Kibana’s visualisation is what is holding ES up for me. I will look into superset+Clickhouse to see if I can come up with a good analysis front for our log data.
(there's a "Forgot Password" link somewhere here).
[do you always look at poster's activity? sounds time consuming.]
But this is far from the possible performance that Logstash offers, so the comparison is at least misleading.
Also, there is no information about the memory usage of the other contestants, which probably don't have memory limits if they don't run on the JVM.
I say all this as someone who stopped using Logstash, in part due to the high requirements - but benchmarks should strive for a fair comparison.
In addition to forwarding to more storage systems (S3, Elasticsearch, syslog, etc), Vector can do things like sampling logs, parsing them, and aggregating them into metrics. Depending on your needs, this makes it easier to reduce your Splunk volume and reduce costs, transition to something like an ELK stack, etc.
We're also working to build up the metrics side of Vector's capabilities. In a way, you can think of Vector as a stream processing system for observability data, capable of feeding into a variety of storage backends.