Systemd.timer, an Alternative to Cron(andrewpillar.com) |
Systemd.timer, an Alternative to Cron(andrewpillar.com) |
Locality of behavior is important. Note how this systemd-thingy is split into two separate files? It's like that a lot in systemd, whoever is doing the architecture doesn't seem to understand why locality of behavior would be desirable, instead taking the IDE approach of writing a bunch of tools to make managing the increased complexity more manageable. Don't glance at a crontab to see when things will next execute, run some other tool that will inspect everything for you. Make things more complex and difficult to read, and then write tools to make it easier to read again.
They could have just added "periodic" service type that accepts timer options, and at least then it wouldn't have to be two files.
Most of the advantages really don't seem like advantages to me. Templated unit files seems especially silly since it's equivalent to just copy and pasting a line in your crontab and changing an argument.
Seems to work for a lot of people, it's just not to my tastes I guess. Still, I find the whole thing pretty mystifying.
Even your own link talks about the Subjectivity of that opinion, and how LoB is often in conflict with DRY and SoC, both of which I support far more than LoB.
In this context systemd favors Separation of Concerns, the the service is what is being run, and the timer is when is it being run...
it is not that the devs do not "seem to understand why locality of behavior would be desirable" it is they disagree that is more desirable than having a logical separation of concerns...
To me I prefer greater separation of concern than locality
I think there's a middle ground. Sure, a single crontab makes it hard for packages to schedule things and makes it hard to assign ownership to individual packages/people/whatever. I can recognize that, and how it would be a problem for a large enterprise or even just for software developers who want to bundle some kind of cronjob with their package.
What you could do instead of a crontab is have one folder with a bunch of services/timers in it, where each file represents a single service that needs to be run. It would mean that you can still understand all of what's going on timer wise by looking carefully at one source, but you can still assign different services/timers to different people/packages/organizational-units/whatever.
Splitting each individual cronjobs into 2 different files is just crazy from where I am. Like you already have a bunch of different service types, at least give us the option to use a "periodic-oneshot" service type instead of this craziness. There's no need to have different packages/people responsible for a service and its timer, and if there is some edge case where it is you can still have ssperate timer services just they can live in a separate periodic-oneshot service file and use the original service as a dependency.
There's separation of concerns and then there's this. Take any principle too far and you get some craziness, and systemd has most of the infrastructure needed to solve this in a much more elegant way. Like at some point it has to just be bad design and not just different priorities, right?
Having option for separate timer is all and well (after all you might want to use it to periodically start service, for example start some sync service after midnight) but for cron-like behaviour it should really have all in one option, perhaps by just adding [Timer] block in .service or vice versa
man journalctl journalctl -u nginx <G to skip to latest, ctrl+C because it's taking too long> journalctl | grep nginx <wait...> go back to man page to look for more options to search
etc
Really feels like "Embrace, extend, extinguish" but the thing they're extinguishing is unix-as-a-programming-environemnt.
A nice feature is that journald gives you flags like "--boot" to see logs only emitted during a specific boot, or "--since '5m ago'" which is not that straightforward to do with the approach you favor. (if there is a way to do it easily, please let me know!)
> ctrl+C because it's taking too long
Try --since=today. Then you end up in less, and that should be more familiar, including for search. This is then the equivalent of traditional daily rotating log files, with no other knowledge needed apart from --since and --unit (the latter is noted in "systemctl status" output as a reminder).
Then every further feature is an optional bonus.
The backing store may be controversial, but the journalctl command line is much more pleasant. "I want to see log messages from the end of the last boot when the system crashed" is no longer a Rube Goldberg command line involving less/sed/grep/cut/awk and manual scrolling, it's supported out of the box with no configuration necessary.
It gets the "one file to define the job" right but that's the only thing that it gets right.
There is no holistic view on the jobs so you can't easily say "show me last jobs that failed" and shove that into alert.
There is email on failure but only on failure so you can't also do simple "if this mail returns fail, mark check as failed, if it returns okay, mark it as okay". You can get that info from logs but you can't get the content of error message of the app from those logs...
Environmental options history is.... interesting, better to just make script that sets everything up beforehand instead of even trying to do it via cron.
From ops perspective of managing multiple servers its utter garbage. From single user perspective it kinda works but is annoying
Systemd's timers should just be [Timer] block inside the .service
With systemd timers, I have to look up how to write a unitfile every. single. time. and it's not a trivial 30-second lookup like every other command, it's a 5-10 minute read through every time. And even then there's weird variations on the timer unit files I don't quite understand, and most of the time it takes more than one try to get right. It fails in some bizarre ways on certain tasks due to internal systemd inconsistencies. (Like using different escape characters in different contexts, wtf?!) And I can't remember how it's set-up so I have to look it up again when I want to administer it or change something and on the whole it's just a huge PITA.
At this point, I've used systemd timers several times MORE than I have cron and I still have no idea how to write a systemd timer unit file without looking it up, and I'd have to start from scratch to write a new one.
About the only positive is that when it fails systemd reports it decently so I can follow up on it, but man, it's not worth it. I've spent a lot more time dicking with the unit files than I have debugging failed cron jobs.
systemd was inspired by launchd (http://0pointer.de/blog/projects/systemd.html), and that does that (https://developer.apple.com/library/archive/documentation/Ma...), so they must have explicitly made that choice.
Speaking of behavior, let's look at cron.
Consider every time you've written one. There's a good chance that it failed for some time because you forgot PATH is make believe there
Or logging. How many times has rotation been rewritten (and broken)?
Without digging through logs that may or may not exist, can you tell me when it last ran successfully?
I can with timers, 'for free'
To me it sounds like proper architecture and your solution would be needless close coupling.
>what if I want it to be started when a request hits a given port?
I really don't think you should be turning your init system into a general programming environment that can do that.
systemd-run --user --unit=timer-name --on-active=60 /exec/path
systemd.services.somehost-sync = {
startAt = "daily";
path = [ pkgs.coreutils pkgs.rsync pkgs.openssh ];
environment = {
HOME = "/home/someuser";
};
script = ''/external/media/sync.sh'';
serviceConfig.User = "someuser";
};
It's impossible. All that opinionated /etc/crontab and /etc/cron.daily shit can go to hell for all I care. Every opinionated distro uses whatever kind of file format that the next best sh guru came up with, which id also prone for exploitation btw.
With systemd there's a failsafe way for maintainers to offer a cross-distro way to integrate their software.
And that's what systemd is about: automation of integration through convention.
I'm still experiencing file corruption regularly with journald on various distros.
Cannot recall corruption nor unreadability with syslog.
systemctl list-timers is vastly more informative in a practical sense than just looking at the crontab. It just straight up tells you exactly when it ran last, how long ago that was, when it will run next, and how much time is left until that happens.
Timers augment the variety of existing service types, so your idea just doesnt work straight up, unless we severely limit the flexibility of what timers can be via the 'periodic' type. Or we make 'periodic-oneshot' 'periodic-serice' et cetera, which feels absurd.
Perhaps we could leave Service= alone & just shoehorn each systemd.timer option into system.service. But then we need some way to view the timer statuses & logs, which we can easily review and handle by having a separate unit. 'list-timers' is ok & might still work but we cant filter and log as well if everything is jammed together. We also cant just disable a timer temporarily, & go manage the service manually for a bit, if somethings going sideways & we need to step in; the two are now one in your world.
And what if you want multiple timers, with different configurations between them (maybe one long running one with WakeSystem= and a shorter one without?). We eliniate a lot of creative ability by glomming stuff together.
I dont like this idea. I think it's a misservice to jam everything together. Systemd has similar patterns of representing different pars in different places that I think has served it well. It's an even more trivial case that is more supportable by this idea you've floated, but there is a systemd.mount and systemd.automoumt. It's just good, as an operator, having clear boundaries between these units, and has always made it clearer to me what aspect Im operating on, and enabled healthier patterns of system growth.
It's not either-or situation.
The .service could just have [Timer] block
Then you could just put all in one file, or still have option of putting timer somewhere else.
Well, that's where you can write a regular timer or just add a dependency to your timer-service.
But also you're sort of getting to the territory of inventing your own scripting language on top of systemd at that point, if your needs are that complex just write a script.
Sure you can, comment that line out of the service file. It is ultimately just a text file that you can edit.
* Running the service on command ( this is the best way to troubleshoot issues with your slightly different environments / cron's version of shell. I've spent many minutes over my career - setting the crontab one minute and the future and waiting for it to run / fail )
* Easily inspecting when the next run of the service is. ( list-timers )
* Straightforward access to the logs ( Always journalctl )
* Specifying environment variables easily
* OnFailure
* Built in functionality for telling when the cron last ran, and whether it was successful or not (so you can do alerting)
* Ability to specify human-readable schedules
* Built-in feature for randomising cron start time to avoid stampeding
If you have a script that you run every 24 hours, what happens when it takes 25 hours to complete?
(My experience is that this happens only years later, and the people who wrote the cronjob are gone.)
Define email-unit-status@.service
[Unit]
Description=Email status for %i to root
After=network.target
[Service]
Type=oneshot
ExecStart=email-unit-status %i
#!/usr/bin/env bash
set -eu
dest="admin@whatever.com"
userflag=
if [[ $HOME == /home/* ]]; then
userflag=--user
fi
html=$(SYSTEMD_COLORS=1 systemctl $userflag status --full "$1" | ansi2html -w -c)
/usr/sbin/sendmail -t <<ERRMAIL
To: $dest
From: systemd <${USER}@${HOSTNAME}>
Subject: SystemD unit failure alert: $1
Content-Transfer-Encoding: 8bit
Content-Type: text/html; charset=UTF-8
Mime-Version: 1.0
$html
ERRMAIL
OnFailure=email-unit-status@%n
Seriously, though, cron has been so utterly reliable for me for so many decades now, it will be really hard for me to ever give it up.
I kid about this, in a way, and I know I should accept the inevitable, but I feel like just moving to Devuan on my laptop and use a nice init system, like OpenRC :D
This sounds a bit weird, but its not easy to get started with, unless you're used to reading man pages. But once you're able to read man pages, all of systemd behavior is documented in the man pages. I do wish there were more human friendly docs though, especially for younger engineers used to better sorts of docs.
Anyways, once you do get the hang of it, its an immensely powerful system, but like any such system it has its quirks and edge cases. You can get 99% of what you need with it though, and for low level tasks on machines, its pretty amazing.
Also systemd documentation isn't compliance with:
* Tutorial
* Explaination
* How To
* Reference
For instance, I want to know which part of systemd uses `SYSTEMD_DEBUG`.
Hold on ... I am digging thru the Git repo.
Screw the systemd manual:
Here's my environment variable reference guide for debugging systemd (that is NOT documented by systemd)
https://egbert.net/blog/articles/debugging-a-service-in-syst...
- from cron: specifying all jobs in one file instead of scattering it across dosens of unit files. In 90% of cases I just want a regular schedule and the command, that's it
- from systemd: mainly monitoring and logging. But also flexible timers, timeouts, resource management, dependencies -- for the remaining 10% of jobs which are a little more complicated
So I implemented a DSL which basically translates a python spec into systemd units -- that way I don't have to remember systemd syntax and manually manage the unit files. At the same time I benefit from the simplicity of having everything in one place.
An extra bonus is that the 'spec' is just normal python code
- you can define variables/functions/loops to avoid copy pasting
- you can use mypy to lint it before applying the changes
- I have multiple computers that share some jobs, so I simply have a 'common.py' file which I import from `computer1.py` and `computer2.py` -- the whole thing is very flexible.
You can read more about it here:
- https://beepb00p.xyz/scheduler.html
- https://github.com/karlicoss/dron#what-does-it-do
I've been using this tool for several year now, with hundreds of different jobs across 3 computers, and it's been working perfectly for me. One of the best quality of life improvements I've done for my personal infrastructure.
Put it in cron, worked first try, and forgot about it.
It looks like systemd.timer will continue this practice ;) I'm expecting someone at Canonical/IBM/RH will be happy to put this as the default scheduler in the next distro release, marking it as a "significant improvement".
Same with the last - templated unit files. Add a parameter or more to the script and you're done.
mkTimer name schedule program
Did that feel cathartic?
This is a trend in SW: facelift every couple of years.
This is a bad thing, not something I’d boast about.
I actually really enjoy using systemd. Being able to start a process with complete isolation without heavyweight docker downloading mystery meat off the internet is a huge boon.
However, one thing systemd is logging. Jounalctl sucks. Grep, cat, etc, work infinitesimally better.
And it's not half-bad, I think, because GRUB2 is just way, way more complicated than most setups need these days.
It's best to shut down any databases prior to this.
Cron requires manual locking, which results in an old service sometimes aborting without cleaning that up, and then refusing to start ever since.
I'm precisely in the middle of an update that's going to throw that out, and just change to a systemd service/timer. Then that problem will go away for good.
Tradition: Just because you've always done it that way doesn't mean it's not incredibly stupid. -- Despair, inc
Every dev and administrator, or office user should have that poster above their desks
It's why I have to check /etc/profile, ~/.profile, /etc/profile.d/*, ~/.bashrc (or whatever shell), /etc/environment, ~/.env, and a few others I don't even remember to figure out why an $ENVVAR is set.
I have to look at two cron-like things to figure out what's scheduled when. systemd-timers has been around and in use for a long while.
It's the same rule I use when purchasing kitchen equipment - often it's better to have a single multi-purpose tool than a bunch of specific implements. Otherwise you end up with a drawer full of strawberry-stem removers, pizza slicers, corn-cob peelers, avocado-slicers, etc.
This is to say nothing of the binary format issues that others have discussed.
I much prefer the systemd solution. As an added bonus, systemd services allow for easy configuration of things like sandboxes and resource limits that would need to be hacked into a giant concatenated command line in cron, presumably without comments even.
With cron this is troublesome, because cron uses an environment different from what you get from the commandline. So things can work from the shell and not from cron.
So to really check that cron works you have to add a line to be called in 5 minutes, then check your mail to see what happened.
With systemd though, you can launch the service at any time and see what it does without all that fiddling.
The whole point is to not need to keep running into a bunch of artisinal handcrafted did-it-myself shell scripts every time come to a system. Being able to compose timers that trigger units enables this nicely with a lot of flexibility. Managing them separately is powerful & makes sense.
(I'm assuming you're writing a bash script on a Linux system)
if grep -qs '/foo/bar' /proc/mounts; then
echo "/foo/bar mounted" echo "/foo/bar not mounted"
return 1 || exit 1
Put these commands below the shebang but before the start of the script. I hope this helps you!
Edit: I added a crapload of newlines to better show the indentation and newlines required for the script to work. Sorry it's ugly
It's bad policy for desktops even if it's good for you in work environments and so you use it anyway. Normalizing that is bad. I said it like a jerk though. Sorry.
It’s not that much different than systemd in that sense.
Especially since the /etc/cron.whatever special rules like files with dots in them get ignored (ugh)
I prefer convention for systemd. The justification for have a separate timer file instead it being in the unit file would be making it easier for someone to quickly look at what services are running on timers vs running by targets.
if timers where in the services file you would have to open each one to find out what had a timer, with a separate file you just look at which service as a timer provided your naming is standardized)
Convention means that behaviour depends on not just what's present but also what's not present. So it is of course missing some of the explicitness. For the directories, you need to understand which directories to look into and in which order they are taken into account (for systemd this is hard enough that's it's best to be tool assisted).
Also, syslog wasn’t capable of logging during the initial stages of boot, which is pretty important.
and makes is hard for admins to schedule things, and make is hard to figure out what is scheduled when, and makes it hard to test if the process will work under the schedule.
I absolutely hate cron's format, and pretty much everything else about it. systemd timers is a WONDERFUL and absolutely needed change to the world of linux administration.
cron was written in a time where verbosity had actual costs to the system, we are not in those times and the need for nostalgia to be cryptic and only use minimum chars is over and I for one am glad for it.
Both in programming and administration people need to be MORE VERBOSE not less
What I am saying is that splitting a basic timer between two different files is really bad for readability and understandability.
you are splitting a service and a timer, not all services need a timer. I think that is point being glossed over
services may be called by a target, a timer, or manually. or all 3 when the need arises.
So for example if I have a process that I need to run on startup, every Friday, and sometimes adhoc I can create a single service file and do all 3, and know it is executed in the same way every time
could not do that with cron or even older init systems
We have :) https://www.freedesktop.org/software/systemd/man/systemd.soc...
No need for multiple anything, the units can be parameterized -- user@my.service, for example
Everything be in 4-digit or 5-digit error code.
Need to RTFM for such codes. Or worse, StackOverflow them.
It's such a weird disconnect seeing people that can't go outside of the mindset that everything is a web application.
Systemd ships their documentation (most of what you asked for, actually) with the package, and their environment variables description can be found in /usr/share/doc/systemd/ENVIRONMENT.md (at least on my machine).
When you're dealing with the applications which form the building blocks of a *nix environment your first step should not be google, but grepping through the /usr/share/doc, and using the apropos and man commands.
I don’t think thats what’s happening here. Providing online documentation thats easy to read, easy to navigate has become quite standard of all kinda of projects and systems. My main programming environment is a mac so Im certainly not gonna have the locally installed docs anyway.
Not only does a good online documentation make it easier to understand for an individual, but it makes it easy to share links etc to others to help them understand too. And the easier it is to comprehend, the better.
> Not only does a good online documentation make it easier to understand for an individual, but it makes it easy to share links etc to others to help them understand too
God forbid developers package their software considering the needs of users that don't have 24/7 internet access. If it wasn't clear from my previous statement, this is what I wanted to express by the "mindset that everything is a web-application".
How to what? systemd umbrella contains a lot of functionality, just like the detractors keep on telling us.
If you found a way to improve the systemd documentation you should probably reach out to the devs: https://github.com/systemd/systemd/tree/main/docs
until it address the 4-ways Divio documentation methods.
If you want people to troubleshoot your stuff (which is arguably the #1 documentation to start with), gotta start someplace (which is for most of us is agrep'ing the code, or save yourself a step and look at my link).
Or the RedHat/IBM systemd development team can quit doing that terse but cryptic error code output.
Or the documentation team can start covering these error codes.
But having to do any form of "agrep" is a sign of poor ... everything, unless formenting job security is what the ultimate goal is.
On my NAS
# strace journalctl -u nginx 2>&1 |grep open |wc -l
827
# strace journalctl -u baked-potato 2>&1 |grep open |wc -l
827
It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful. If the binary database was a SQLite file with columns containing app name/log level/etc. it would actually be useful but current one is just shit.
Bonus round:
#:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
#:/var/log/journal time journalctl -u nginx >/dev/null 2>&1
real 0m3,232s
user 0m0,036s
sys 0m0,292s
#:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
#:/var/log/journal time ag -R nginx . >/dev/null 2>&1
real 0m2,531s
user 0m0,004s
sys 0m0,290s
> It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful.
That sounds interesting, can you please elaborate on the internal structure of journald files or link to further documentation? And why would I care if journald handles it for me?
Well, I was routinely getting multi-second wait for operations like systemctl status servicename which is not something rare, because it didn't even kept the pointer to file that has last few lines of app's logs. It's not just benchmarks
It also trashes cache with hundreds of MBs of logs instead of stuff apps running on server actually needs. An equivalent of "tail -f" can go thru hundreds of megabytes of binary logs.
https://systemd.io/JOURNAL_FILE_FORMAT/
On the other hand I agree that journald seems to call open syscall more often than I would originally expect, which can be a problem for some edge cases[1], but I don't consider this to be a real problem.
The difference is that you can use ls to find out, the same ls that you use when working with everything else, rather than needing to know some journald-specific thing that will only ever work for journald and will no doubt change again in another 5 years.
It's a big difference, not because one command, but because tail is part of the standard unix toolset that works with all files. Except now you have this one log file that is special and needs a different unique tool.
I do. I specifically want that as a more valuable and powerful feature. Undifferentiated, open-ended, general purpose, sharp tools are infinitely more useful.
If cache is hot, you have hundreds of MBs of systemd logs littering your buffer cache, while it could be used for something more useful
If cache is cold simple operations like systemctl status take up to tens of seconds, especially on loaded servers. Example:
# time systemctl status influxdb >/dev/null
real 0m11.913s
user 0m0.013s
sys 0m0.531s
Your example with dropping cache and running systemctl status is indeed interesting, 11s looks like too much. But it's a number without a context, and I wonder how big a problem this actually is. I haven't noticed it myself before.
> It doesn't even keep index of "last file where app wrote logs" which causes above.
While I definitely see some room for optimization, I'm not sure what you mean here: journald uses one active journald database file, there should not be a problem with figuring out where the file is, should it?