I have a bunch of Python scripts running on different servers, some of them scheduled as recurring cron jobs.

Now I'd like to have a central place to collect status messages from all of these "services".

Basically, I want to sprinkle snippets into my scripts that say, "If X happens, notify my dashboard", either as an error message ("Crashed, OOM"), or a success message ("35 items added to database".

Same for when I run a command on my terminal that takes a long time.

Then I'd like a dashboard that shows all separate services in a table. Green for those that work, red for those that don't. And the ability to drill down into the log of status messages over time for each of them.

The dashboard should also be able to configured such that a worker will be considered "dead" if it hasn't posted any status message within the last n hours. In an ideal world, I wouldn't even need to set this up on the monitoring server itself. Instead, I'd just pass the maximum time window allowed between messages along with the message itself:

  Dashboard.log(name="WeatherBot", success=true, message=f"{n} items added", max_seconds_until_next=24*60*60)

I'd also like phone notifications.

However, it's important to me that I can choose to mute notifications for any specific type of message. I want to be able to say, "Don't notify me about this particular thing anymore, but please continue to keep a log, and please do continue to notify me about everything else".

Getting the message that my new service is running successfully is great for the first two or three days. But then then info becomes noise, and I'd now rather be notified only when something goes wrong.

So far, I've been using Prowl (similar to Pushover). But I'm afraid of using it too much, because I have no way of disabling the notifications from the place where I receive them (my phone). Instead, if a notification starts to get annoying, I need to log in to the server and disable the Pushover code there (provided it isn't a running process). Also, just because I don't want notifications about an event right now doesn't mean I don't want to still collect a log of these status messages, which I can inspect at my leisure.

When researching this, I've found services like Gatus [0] or Statping [1]. The latter, so far, seems closest to what I'm looking for, though it doesn't quite fill my requirements.

I was thinking about building it myself. But it seems such a common problem that I'd be surprised if a solution like that wasn't already out there.

Thanks!

[0] https://github.com/TwiN/gatus [1] https://docs.statping.com/#doc-general-notes

user:	IceCoffe
created:	December 20, 2019
karma:	7