It seems nobody else cares about this use case and wants you to use LogQL and the incredibly clunky Grafana web UI instead, because it makes it possible to aggregate across many different processes, slice and dice by various labels, etc., which as I said, I have never (or almost never) actually wanted to do.
Hopefully this new UI is a step in the right direction as people won't need to futz around with LogQL anymore, but it seems like it still doesn't quite do what I want.
Is this google cloud logging? If so, personally I quite like it, especially for looking through logs from multiple sources at the same time. Being able to put all your logs through there, and then search them with a simple query language, feels very convenient.
Then if I know which pod I'll filter down to a specific pod with `{pod="workload-foo-1234"}`, sometimes I'll search for a specific term (error message etc) with `{pod="workload-foo-1234"} |= "error message"` then look at the logs around that. There's really no point writing complicated queries unless you need to.
I have one log file per day per host on my syslog server and can use "sort" to order across multiple files.
Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.
It really isn't. It's a single binary with a relatively simple configuration file, you throw logs at it via an API (which a bunch of logging agents support, and syslogs can be sent to it).
Then the actual queries aren't all that complex, it's just a difference of cd-ing to the correct folder for the date/server to be able to cat and grep vs writing a query that selects by server name and filters by date.
The learning curve and maintenance of Loki are quite minimal, but the value add is quite significant in most cases. Being able to do cross-host queries, metrics from logs (how many times did error X occur in the logs), as well as easy visualisations is pretty useful.
If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.
Distributed anything is actually easier in most cases because you will always have input and output. Sure, if you're debugging a complicated and coordinated "dance" between two concurrent threads/processes then yeah fully agreed, but then you're deep in uncharted territory and you need all the help you can get.
This is still a valid use case but pretend for a minute you have thousands or millions of log lines to inspect. Even after filtering for ERROR level only, you still have too many "those are normal" errors, devs swear (but do not fix). And maybe the data you need to diagnose isn't even in ERROR!
The solution? Use log queries to compare a normal and abnormal process or cluster, group them by some kind of fingerprint, then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal. This lets me rapidly identify problems at scale that would otherwise take hours of pouring through logs to exclude stuff by hand.
This works any time you can divide logs into "good" and "bad." Example scenarios:
- canary analysis, comparing canary and baseline
- single faulty pod in a deploy, comparing the bad container to the n good ones
- one AZ or region in a multi-region deploy
- now versus yesterday, or versus an hour ago, etc
- Android versus iPhone
> then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal
That's the rub that I think you are missing. In distributed and/or cloud environments it is quite unusual for there to be a single end-to-end process, and thus we need new ways to trace across a system.
In harmony with tracing, we also need the aggregated view _across_ the estate to understand where system hotspots, levels of throughput, redundant infrastructure, error rates, etc.
Kibana is ok (definitely beats grep) when you want to look across a fleet and determine if a specific thing is happening. But when you have a specific symptom that happens on a particular instance, what you want to do is see logs in the order they happened, and Kibana isn't close. Querying and viewing logs are just slow and cumbersome relative to less/grep.
There is a CLI tool with the same name that does something similar - https://github.com/johanhaleby/kubetail
select * from stdout, stderr
where session_id = 123456
You can also do something like
select client_ip from requests where elapsed_ms > 10000
which is incredibly powerful
But these log query systems can also optimize these queries for instance by by sampling, using distributed trace ids to ensure you get shown corresponding, allowing you to get only logs where at least one step in the trace errored, etc.
ldpreloadhook has an example of hooking write() with LD_PRELOAD=, which e.g. golang programs built without libc don't support.
When systemd is /sbin/init, it owns all subprocess' file handles already, so there's no need to close(0), time, open(0) with gdb.
Without having to logship (copy buffers that are flushed and/or have newline characters in the stream) to a network or local Arrow database files and or SQLite vtables,
journalctl (journald) supports pattern matching with: -t syslogidentifier, -u unit; and -g grepexpr of the MESSAGE= field:
journalctl -u <TAB>
journalctl -u init.scope --reverse
journalctl -u unit.scope -g "Reached target" # and then "/sleep" to search and highlight with less
journalctl -u auditd.service
# this is slow because it's a full table scan, because
# journald does not index the logfiles;
# and -g/--grep is case insensitive if the query is all lowercase:
journalctl -g avc --reverse
journalctl -g AVC --reverse
# this is faster:
journalctl -t audit -g AVC -r
# this is still faster,
# because it only searches the current boot:
journalctl -b 0 -t audit -g AVC
# these are equivalent:
journalctl -b 0 --dmesg -t kernel
journalctl -k
#
journalctl -b 0 --user | grep -i -C "xyz123"
There is a Grafana Loki Docker Driver that logships from all containers visible on that DOCKER_HOST docker socket to Grafana for querying with Loki: https://grafana.com/docs/loki/latest/send-data/docker-driver...
Podman with Systemd doesn't need the Grafana Docker Driver (or other logshippers like logstash, loggly, or fluentd) because systemd spawns containers and optionally pipes their stdout/stderr logs to journald.
Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor. Chronograf is their WebUI which provides a query interface for configurable chart dashboards and InfluxQL.
Grafana supports SQL, PromQL, InfluxQL, and LogQL.
Graylog2 also indexes logfiles.
But you can't query stdout and stderr you or /sbin/init haven't logged to a file.
I do a fair amount of tracking down of issues with LogQL. Looking for logs specific to a customer support ticket. Filtering for logs by a traceId for distributed traces.
I have serious doubts this new UI is something I will care about at all.
The explore ui for setting labels is atrocious and painful, and I'd rather just give me the text input for LogQL*
*: Please FFS someone fix the Ctrl+f creating a vscode like find dialog that only finds inside the text input. I never want to do a find specifically isolated to my LogQL
Quick update: that has been fixed and will be available in Grafana 11.1.
My next task is to get centralized logging going with Victoria Logs + Vector, I'll have to check this out once I get everything setup. I believe I can use LogQL with Victoria Logs but I haven't tried it out yet. https://docs.victoriametrics.com/victorialogs/logsql/
I'm kind of amazed the UI doesn't select small time ranges iteratively to build up the response, especially since I believe this is what the CLI does. Perhaps this is also part of their cloud offering provides and it is part of their marketing strategy. Not a good one because if we came down to the decision I would start by looking for something else from being p'ed off by Loki.
But I guess it still works pretty well considering it is free.
Also Loki on the backend splits/parallelizes requests if it can.
The Grafana backend Mimir / Loki / Tempo products all appear to be architected pretty similar, and I'm more experienced operating Mimir, but the answer to read load often has to just do with right-sizing the deployment scale, and using caches aggressively.
Unless you have the bank to go with their managed service, don't bother.
I want to query my logs like a SQL table, not a time series database.
grafana is very quirky for searching for traces. And has a huge learning curve.
What's the TL;DR for why Datadog is better?
I will stop wasting my time here and try the Metric Explore panel
Somehow, VictoriaMetrics manages to provide much better results.
https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
https://github.com/nklmilojevic/home/tree/main/kubernetes/ap...
Here you have Vector in aggregator + agent mode and several sources. VictoriaLogs also recently added Grafana datasource so it is fairly easy to set it up:
https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...
I'm a big fan of VictoriaMetrics as well and we use it extensively in my company at high scale.
Wrinkle: two docker remote logging plugins I tried (e.g. loki, elastic) didn't seem to work on ARM processors out of the box.
I use Podman for all of my container stuff and there are issues with how Podman produces JSON logs
https://github.com/vectordotdev/vector/issues/6807 https://github.com/containers/podman/issues/16317
which needs to get fixed before I can use it for my workloads.
Since Grafana built Loki, it doesn't make any sense why Grafana would create a separate querying UI app for Loki when they already have Grafana Explore. Prometheus (and I assume its UI) was created by Google [edit: sorry, created by SoundCloud, inspired by a Google Borg tool] before Grafana became the de facto Prometheus query UI, so it’s not really analogous.
The current generation of observability tools is built for distributed systems that are basically too complex to reason about, and so you have other ways of monitoring and debugging them. When you have 10's of k's of ephemeral containers running hundreds of services, you can't just look at some logs for a server to understand what's going on (ignoring the fact that servers aren't even a primitive in this system).
10's of GBs of logs a day just doesn't move the needle on pricing. They want the customers that are going to generate 7 figures in revenue and those customers aren't talking about aggregating logs from a few hundred servers.
Admitably I learned how to use basic tools 25 years ago, but that's an investment that can be used for decades.
cat *web*log | "grep 34.5.22.4" | sort -n | less
Will whatever today's favoured log query/filter/etc be around in 25 years? Last time I looked at this people were going on about logstash and elasticsearch. Nobody could show me how to to the above command without touching the mouse.
Now sure, cat and grep can be sluggish on millions of lines (which is the main reason I'm tempted by loki or similar), and there's always some twat that comes along with "useless use of cat" [0], but the kind of pipeline processing serves me well and it seems a very different way to think about things when you need to access things from a database. Maybe I'm in a local maximum, but it's good-enough for me to find out what's going on.
[0] https://stackoverflow.com/questions/11710552/useless-use-of-...
You have to connect to your server, get to the correct folder, and then run the cats and greps which are easy (if you have to do some more advanced filtering with awk it gets more complicated.)
Connecting to Grafana and running a simple label query is practically the same in terms of complexity and time, but with vastly more features available.
> Will whatever today's favoured log query/filter/etc be around in 25 years? Last time I looked at this people were going on about logstash and elasticsearch. Nobody could show me how to to the above command without touching the mouse.
You can run ElasticSearch queries via the API, and can still do it today. I don't know about, but Loki is a statically compiled binary with only optional external dependencies. You'd still be able to run it in 25 year just fine.
Logs often contain privileged info, if not reveal a bit about how the application behaves. It is risky to do that.
I pulled this technique from canary analysis and applied it to production outage analysis. In canary, you have a guaranteed random stable population that lets you perform accurate comparisons. Elsewhere, we can try to make that assumption but it might break down. For example, regional holidays can radically alter customer behavior over time or between regions. So it's not perfect but it's often good enough to provide me insights while on call.
And, it requires advanced log queries to perform all these filtering, grouping, counting and scoring functions.
Best to configure view for this to limit columns and maybe pre-configure some filters. Plus annotate your logs with timestamp, so you rely on time of event and not time of ingestion.
But these both are one-time configuration thing and then you can simply scroll.
What's missing is that I don't want to learn and use some clunky web UI in order to do this. I want the UI to be "download this text file" and then use the tools I already know and understand (local text processing utilities and text editors)
If you mean one instance in each pod, then each should be labelled differently and you can filter down to one instance.
If you mean running multiple instances in each pod (and container?), then the standard kubectl log output will also have them all joined together. For both of those, you would need to add another unique identifier to each line, or run each instance in a separate container so you can submit the logs with the pod name and container name combined being the unique identifier.
What tools are you using? For me it's often just grep and awk with temp files, maybe a touch of python occasionally.
Not being able to filter between multiple processes or multiple restart of a container was a genuine issue, not being able to filter between pods of a deployment is not.
Anyhow, the nice thing about the system is that one can get anything that is preferable as long as the logs are annotated correctly (with pod and container id).