Show HN: Simple Log Alerts to Slack

Show HN: Simple Log Alerts to Slack(profitview.net)

26 points by tradrich 3 years ago | 14 comments

There are many log alerting systems on the market. The best known is probably Datadog. There’s also Logtail, Papertrail, Splunk, Logstash and others.

These are well put together products with a host of great features, such as excellent UIs, sophisticated live searching via web interfaces and sometimes query languages and alerting. They require various levels of installation and they have costs, either through volume-based tiered systems or monthly payments.

For a bootstrapped business, this can be problematic, for instance when a surge of logs - indicating a possible important problem that needs to be solved - pushes volume on to another tier. Should the “log ransom” be paid?

Instead, I recalled from earlier times surely the simplest log watcher: Swatchdog [1]. It is rather venerable software. Its file history from its source download shows dates in 2015, but it was written much earlier - the 90s or possibly 80s by Todd Atkins [2].

We wanted to have alerts in Slack - the blog explains how we did it. In short: *very simply*. The code is available [3].

[1]: https://github.com/ToddAtkins/swatchdog

[2]: https://www.linkedin.com/in/toddatkins/

[3]: https://github.com/profitviews/swatchdog

PuffinBlue 3 years ago |

This looks very interesting. It seems to work on a regex match, so you'd set what you want to see alerted on and then you receive alerts if that match is made?

I like the opposite method - default alert on everything and develop an allowlist that quietens things down you don't want to hear. This is great for alerting you to unexpected things. And once in a while you actually want to know about some of those things :-)

It may sound very noisy but it's not too bad, especially once you're allowlist is setup. Logcheck[0] is a good tool for this and it runs by default at 2 minutes past each hour, emailing in a report of everything that isn't allowed. I think it matches some regex to what it deems higher threat events and those are always alerted on.

I'll conceed that this method isn't stellar for cattle! And we don't bother with it for things like kubernetes clusters or servers with semi-regular turnover for instance.

For pets and long lived servers that need looking after it's a good tool.

[0]https://logcheck.org/

Jedd 3 years ago |

An interesting approach, especially the hybrid python + perl bundling. Embrace the 'you can't please everyone, all of the time' mantra.

At work we're using a combination of Splunk, Microsoft Sentinel, and Grafana Loki (can really recommend the latter combined with promtail).

There's some curious artefacts in this repo - the systemd service file calling usr/bin/env to a $PYTHON_EXECUTABLE (set in the env file to /usr/bin/python3) but the python script being called starts with '#!/usr/bin/env python' (which on my Debian unstable system launches python2).

Throwing executables under /etc/ is a .. contentious manoeuvre.

And the - so far as I can see - lack of support for handling journald output I suppose is a reflection of this sentiment:

> There’s endless criticism of systemd in the Linux community and I expect it will be properly superseded in the next few years. [...] It’s a curiously un-robust system.

slyall 3 years ago |

You could probably also use fluentd to do the same thing. It can process the logs locally and sent alerts instead of forwarding logs to a central source.

How to grep the logs for errors and send an email

https://docs.fluentd.org/how-to-guides/splunk-like-grep-and-...

The slack plugin you could use instead of email:

https://github.com/sowawa/fluent-plugin-slack

giraffer 3 years ago |

If most of your stack is written in Python you could also consider setting up a custom log handler (with additional filters or formatters) such as https://pypi.org/project/slacker-log-handler/

The Python logging framework is quite flexible

NovemberWhiskey 3 years ago |

Hrmm, I'm not sure I'd call Datadog or Splunk "log alerting systems". That's one of n capabilities that involves logs in each of those platforms. If you just want to fire an alert based on a match in a log line, you definitely do not need those platforms, I agree.

iroddis 3 years ago |

I really like this, the simplicity is really refreshing.

Nothing to do with the article, but it would be nice if Slack (or IOS/Android) offered better customization of notifications. Errors and warnings are indistinguishable from regular messages on mobile, which means they often get lost in the shuffle.

yabones 3 years ago |

Very nice. I think a lot of admins jump straight to the most complicated Elasticsearch/Datadog/NewRelic stack and don't even take a second to build something simpler.

For "unfussy" machines, I like to just set up rsyslog to send high priority syslogs right to email. [1]

mxuribe 3 years ago | |

I agree; I like simpler approaches first. Then - if needs grow - one can grow/evolve to more complxity, but only if even needed.

Also, I used to use emails as notifications for many, many years...but lately have been using my own little matrix cli app/script[1]. I'm such a fan of matrix nowadays. Also, I stopped sending a full log file, and merely send only a snippet...Often, its enough for me to get the gist of what is happening. And, if certain thresholds exceed beyond crazy levels, then the messages become more urgent from my machines. :-)

[1] shameless plug: https://github.com/mxuribe/howler

jesterson 3 years ago |

Not trying to break your party, but GrayLog2 solves all those problems. Most businesses/startups do not need well marketed DataDog or Logtail.

Free, fast, versatile, adaptable to almost every case

tradrich 3 years ago | |

http://graylog2.org/ takes me to graylog.org whose pricing is not free and not cheap.

Your company?

Anyway - not a "competitor" to the genuinely free and open-source Swatchdog.

jesterson 3 years ago | | |

You may want to look a bit deeper than pricing page. And no, it's not my company.

Not sure what "genuinely free" means.

troysk 3 years ago |

Why use Slack? Slack has rate limits and one will not get the notifications when something real goes down as there will be 100s of messages overloading Slack API.

tradrich 3 years ago | |

Slack is convenient just because we use it for some other purposes.

Someone else pointed out that Telegram is also a good idea - that's very easy to do. I've set up Telegram alerts for other purposes.

troysk 3 years ago | | |

Rate limits are present in chat apps to prevent the service from being abused. Notification tools don't have that.