One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since.

Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into.

But, if you're doing a 'useless' use of cat, you're probably just doing it in an interactive session. You're not writing a script. (Or maybe you are, but even still, I bet that script isn't running thousands of times per second. And if it is, ok, time to question it).

So you're wasting a few clock cycles. The computer is doing a few billion of these per second? By the time you explain the 'useless' use of cat to someone, the time you wasted explaining to them why they are wrong, is greater than the total time that their lifetime usage of cat was going to waste.

There's a set of people who correct the same three pairs of homophones that get used incorrectly, but don't know what the word 'homophone' is. (Har har, they're/their/there). I always liken the people who are so quick to chew someone out for using cat, in the same batch of people who do this: what if I just want to use cat because it makes my command easier to edit? I can click up, warp to the front of the line, and change it real quick.

Sorry. I did say, it is a pet peeve.

  <f a | b | c

  cat file.txt \
  | sed '1s/^\xEF\xBB\xBF//' `# Strip UTF-8 BOM at the beginning of the file` \
  | ...

   cat x | cmd
   < x cmd

  cat file.log | tr 'a' 'b'

  tr 'a' 'b' file.log

   tr a b < file.log

   < file.log tr a b

My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

One thing I've done to identify infrequent log entries within a log file is to remove all numbers from a file and print out a frequency of each. Basically just helps to disregard timestamps (not just at the beginning of the line), line numbers, etc.

  cat file.log | sed 's/[0-9]//g' | sort | uniq -c | sort -nr

This has been incredibly helpful in quickly resolving outages more than once.

    alias numberless="pbpaste | sed 's/[0-9]//g' | pbcopy"

My tips:

1) Fuck grep, use ripgrep, especially if you have to scour over an entire directory.

2) Get good with regex, seriously, it will shave hours off your searching.

3) For whatever application you are using, get to know how the logging is created. Find the methods used where said logs are made, and understand why such a log line exists.

4) Get good with piping into awk if needed if you need some nice readable output.

Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good.

The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly. For Java, log4j can use class in your logging pattern, and that plus log level is usually pretty specific.

Once you have a fingerprint, the rest is just counting and division. Over a specific time window, count the number of log events, for every finger print, for both good and bad systems. Then score every fingerprint as (1+ # of bad events) / (1 + # of good events) and everything at the top is most strongly bad. And the more often its logged, the further up it will be. No more lecturing people about "correct" interpretations of ERROR vs INFO vs DEBUG. No more "this ERROR is always logged, even during normal operations".

One thing I didn't see was how to use GREP to view the lines before and after a match:

   grep regex /var/log/logfile -A5 #To view the next 5 lines
   grep regex /var/log/logfile -B5 #To view the previous 5 lines
   grep regex /var/log/logfile -05 #To view the 5 lines before *and* after the match

This is super handy to find out what happened just before a service crashed, for example.

    grep regex /var/log/logfile -05 #To view the 5 lines before *and* after the match

Loosely related: a few years ago I wanted a simpler alternative to some of the more feature-full log viewers out there so I threw together a tiny (50kb) app that might be useful to some folks in here.

All it does is consistently colors the first field in a line from stdin so you can quickly see which log lines have the same first field.

I used it in combination with the parallel[0] command to prefix log lines by replica name when tailing logs across machines: https://github.com/jasisk/color-prefix-pipe

[0]: https://www.gnu.org/software/parallel/

As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful.

And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers because of logs" just died when all logs were accessible via one web interface. I added a log aggregation and suddenly only ops needed access to servers.

Designing that thing with accessibility and discoverability in mind is a whone nother topic though.

Glad to see lnav (https://lnav.org) already getting some love in the comments. Hands-down the most reliable source of "thank you! I wish I'd known about this tool sooner!" responses, even from v experienced sysadmins / SREs.

A minor optimization is collapsing the grep -v, from this:

  cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4

to this:

  egrep -v 'THING1|THING2|THING3|THING4' file

That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep.

A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option.

> you’ll get overwhelmed by a million irrelevant messages because the log level is set to INFO

I know this happens, but I think it's because programmers are abusing INFO. In principle it's reserved for messages that are informative at a level sys admins and a few others can make sense of and use. Unfortunately abuse often leads to "We turned INFO off" making it much harder to diagnose things after the fact.

When I was training sysadmins back in the dark ages, one of the rules I taught was: know what good looks like in your logs. If you are scanning hundreds of lines of logging under duress to find a smoking gun, and you don't know the difference between what the logs normally show, and what you are seeing, you'll waste a lot of time.

Corollary is that good day logs should be minimal and "clean", e.g not logging a lot, or, logging nice and predictably (which makes them easy to strip out via grep -v, etc.)

> Often log lines will include a request ID.

Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

I recently used clickhouse-local to do some log analysis on a lot of elastic load balancer logs (~10s of GBs) and it was spectacular.

In short, you can add clickhouse-local to a shell pipeline and then run SQL queries on the data. An example from the docs:

$ ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' | clickhouse-local --structure "user String, mem Float64" --query "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty"

I wrote https://github.com/ljw1004/seaoflogs - an interactive filtering tool, for similar ends to what's described here. I wrote it because my team was struggling to analyze LSP logs (that's the protocol used by VSCode to communicate with language servers). But I made it general-purpose able to analyze more log formats too - for instance, we want to correlate LSP logs with server logs and other traffic logs.

(1) I wanted something where colleagues could easily share links in workplace chat with each other, so we could cooperatively investigate bugs.

(2) For LSP we're often concerned with responsiveness, and I thought the best way to indicate times when viewing a log is with whitespace gaps between log messages in proportion to their time gap.

(3) For LSP we have lots of interleaved activity going on, and I wanted to have visual "threads" connecting related logs.

(4) As the post and lnav say, interactivity is everything. I tried to take it a step further with (1) javascript, (2) playground-style updates as you type, (3) autocomplete which "learns" what fields are available from structured logs.

My tool runs all in the browser. (I spent effort figuring out how people can distribute it safely and use it for their own confidential logs too). It's fast enough up to about 10k lines of logs.

    Uncaught TypeError: [].filter(...).reverse()[0] is undefined
        onload https://ljw1004.github.io/seaoflogs/demo.html:14

Some extra tips:

  Keep access logs, both when a service receives a request and finishes a request.
  Record request duration.
  Always rotate logs.
  Ingest logs into a central store if possible.
  Ingest exceptions into a central store if possible.
  Always use UTC everywhere in infra.
  Make sure all (semantic) lines in a log file contain a timestamp.
  Include thread ids if it makes sense to.
  It's useful to log unix timestamp alongside human readable time because it is trivially sortable.
  Use head/tail to test a command before running it on a large log file.

If you find yourself going to logs for time series data then it is definitely time to use a time series database. If you can't do that, at least write a `/private/stats` handler that displays in memory histograms/counters/gauges of relevant data.

Know the difference between stderr and stdout and how to manipulate them on the command line (2>/dev/null is invaluable, 2>&1 is useful), use them appropriately for script output.

Use atop, it makes debugging machine level/resource problems 10 fold easier.

Have a general knowledge of log files (sometimes /var/log/syslog will tell you exactly your problem, often in red colored text).

If you keep around a list of relevant hostnames:

  cat $hostname_list_file | xargs -P $parallelness -I XHOSTNAME ssh XHOSTNAME -- grep <request_id> <log_file>

This needs to be used carefully and deliberately. This is the style of command that can test your backups. This style command has caused multiple _major_ outages. With it, you can find a needle in a haystack across an entire fleet of machines quickly and trivially. If you need to do more complex things, `bash -c` can be the command sent to ssh.

I've had an unreasonable amount of success opening up log files in vim and using vim to explore and operate on them. You can do command line actions one at a time (:!$bash_cmd), and you can trivially undo (or redo) anything to the logs. Searching and sorting, line jumping, pagedown/up, etc, diffing, jump to top of file or bottom, status bar telling you how far you are into a file or how many lines it has without having to wc -l, etc.

Lastly, it's great to think of the command line in terms of map and reduce. `sed` is a mapping command, `grep` is a reducing command. Awk is frequently used for either mapping or reducing.

also, journalctl. Very nice to have all logs in one place in a semi-structured nature

`journalctl -u myservice -S "5 min ago"`

Surprised to see that under the section "correlate between different systems" tracing isn't mentioned as an alternative approach. That's what tracing is: logging across different systems and getting that structure all stitched together for you.

“scroll through the log really fast”

This is a great point and probably underrated.

You may also benefit by scrolling sideways really fast.

https://en.m.wikipedia.org/wiki/Superior_colliculus

Best tips I discovered: use emojis. They have colors and they are easy to spot.

For instance if an API call is made use the phone emoji, when there is a timeout use a clock, when an order is dispatched used a package...

When you have to go through huge log file it is a life saver.

A few weeks ago I had a windows installer that was silently failing when upgrading from an older version (installation from scratch was working without issues). And as windows install logs aren't exactly easy to read, I was stumped, until I took an upgrade log from an older, working build, strip all date information from both files and compare them, checking all the sections which were different, until I found a line indicating that a colleague had forgotten about a limitation when dealing with msp (don't delete components on minor upgrades, but I didn't throw any stones as I've done the same mistake, twice, one and two years ago...)

One of my favorite tricks is to use a visual difftool.

Copy good log into left panel. Copy bad log into right panel. Quickly show which lines are new, which are missing and which are out of order. Obviously ignore the timestamps ;)

Just use structured logs, its 2022 ffs.

> cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4

Keyboard manufacturers HATE this simple trick:

grep -vE "THING1|THING2|THING3|THING4" file

Looks great, but I struggle to get this working with the two main log sources on my system:

1) systemd logs (binary data stored at /var/log/journal)

2) docker logs (from various containers running on the host)

Any hints on getting this working? I don't see this covered in the docs (https://docs.lnav.org/en/latest/formats.html).

    journalctl -o json | lnav

Thanks for linking to this!

Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild.

--

[0] - https://github.com/doublep/logview

At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking things in prod, we haven't tried experimenting with much else.

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.

Yes! lnav (https://lnav.org) is phenomenal. Embedded SQLite... easily scriptable... OOB log formats galore, or define your own... it's a mini ETL powertool that scales to at least a few million rows and runs in your terminal. Maintainer's a friendly dude, too.

Interesting! Do you know how it compares to multitail? (https://www.vanheusden.com/multitail/) They look very similar.

    $ multitail /var/log/install.log -I /var/log/system.log

    $ lnav /var/log/install.log /var/log/system.log

  * Monitoring files/directories for updates
  * Decompressed files/archives
  * Detected the log format for each file
  * Created SQLite vtables that provide access to log messages
  * Built an index of all the log messages in all the files so
    you can jump to a certain point in time or to the
    next/previous error message.
  * Display all log messages with syntax highlighting

Our Easy Data Transform software is intended for data wrangling and desktop ETL. But it has lots of features useful for browsing and manipulating log files, including:

* powerful filtering (including with regex)

* smooth scrolling of millions of rows of data

* support for csv, text, xml, json, Excel and other formats

* available for Window and Mac

Came here to say the same - massive props to the lnav devs.

> A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option.

'pcregrep' still exists. But with PCRE2 supplanting PCRE, it is now spelled 'pcre2grep'.

I don't know the precise history of 'grep -P' and whether 'pcregrep' was actually absorbed into it, but 'pcregrep' is its own thing with its own features. For example, it has a -M/--multiline flag that no standard grep (that I'm aware of) has. (Although there are some work-arounds, e.g., by treating NUL as the line terminator via the -z/--null-data flag in GNU grep.)

  # pcre2grep
  bash: pcre2grep: command not found...

  # yum install pcre2grep
  Last metadata expiration check: 1:58:58 ago on Tue 13 Dec 2022 11:45:44 AM CST.
  No match for argument: pcre2grep
  Error: Unable to find a match: pcre2grep

  # yum whatprovides pcre2grep
  Last metadata expiration check: 2:09:25 ago on Tue 13 Dec 2022 11:45:44 AM CST.
  Error: No matches found.

  # rpm -qa | grep pcre2 | sort
  pcre2-10.40-2.0.2.el9.x86_64
  pcre2-syntax-10.40-2.0.2.el9.noarch
  pcre2-utf32-10.40-2.0.2.el9.x86_64

  # which grep
  /usr/bin/grep
  # ldd /usr/bin/grep | grep pcre
   libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efc473c4000)

I once compared the speed of these two approaches, rather accidentally. I did output colorization by adding ANSII sequences. I thought, of course one process must be more efficient than a pipe of processes. After the rewrite, I was disappointed about the slowdown and reverted back to the pipe.

PS I checked back and I used sed rather than grep. I think the result would hold for grep but the morale is that you should verify rather than assume.

I have around 50 seds in the pipe, running in parallel (which is what makes it faster), it would have been a half of that when I tried the rewrite.

I usually prefer to pipe fgrep -v into fgrep -v. With egrep you need to escape brackets and other characters.

  $ fgrep -h
  fgrep: warning: fgrep is obsolescent; using grep -F
  $ egrep -h
  egrep: warning: egrep is obsolescent; using grep -E

  $ cat ~/fmgrep
  #!/bin/awk -f

  BEGIN { split(ARGV[1], s, ","); ARGV[1]="" }

  { m = 1; for(c in s) if(index($0, s[c])) m = 0; if(m) print }

  $ ~/fmgrep sshd,systemd,polkitd /var/log/secure

Or the following if pipes or egrep make you nervous:

grep -v -e THING1 -e THING2 -e THING3 -e THING4 file

Some of these are KPIs (Key Performance Indicators). What we did at a previous job was to have a system like Etsy's statd [1] (it's an easy system to implement) and it made it easy to add statistics like latency of requests, number of errors, just about anything that could be measured, without excessive overhead (in terms of source code).

[1] https://www.datadoghq.com/blog/statsd/

> Always use UTC everywhere in infra

Can Amazon do this? They use UTC and your local browser’s time seemingly randomly depending on AWS service, and it drives me nuts. They usually (not always) put the timezone next to it, but why can’t they just have a mandate that it either is or is not UTC?! (The worst one is that the Lambda console is UTC but cloudwatch isn’t, so you think you haven’t received a request in hours but then you did)