Avoiding Complexity with Systemd(mgdm.net) |
Avoiding Complexity with Systemd(mgdm.net) |
> systemd provides ways to restrict the parts of the filesystem the service can see.
So like chroot and namespaces? Why do I have to depend on systemd when these are native features provided by Linux?
So systemd provides a friendlier abstraction of these concepts. Great, but so do Docker and Podman and many other tools that can actually be installed without taking over the rest of the system.
Having your application actually use systemd libraries further increases this dependency and makes it no longer usable but on a subset of Linux machines. This would be fine for some controlled production deployment, but is awful for usability and adoption.
Not like namespaces - using namespaces. And for the same reason we use other high level abstractions and high level languages rather than handcrafted assembly. You don't have to depend on it either - you can still use chroot instead of you want, but it's more work that way.
> Great, but so do Docker and Podman and many other tools that can actually be installed without taking over the rest of the system.
Docker installs a service which takes over lifecycle management, restarts, and traffic proxying for apps. It injects and managed multiple firewall chains. It pretty much takes over network management. And it's still stuck on the old cgroups format so it forces that on your system. It really doesn't win this comparison.
> Having your application actually use systemd libraries
You don't need them. Everything from the post is defined in simple environment variables. For example socket activation is maybe 3 extra lines when done from scratch.
How so? If I need filesystem isolation, I'll use the simplest tool that provides it. In this case probably a container runtime. Note that none of your criticism about Docker applies to Podman.
Why would I ever want to use containers with a tool that forces (OK, strongly suggests...) me to use it as an init system, logging system, network manager, DNS resolver, and whatever other aspect of my Linux system its authors think it should manage?
I apologize for retreading the same discussion on this topic, but like others mentioned, adopting an incredibly complex tool doesn't mean you're simplifying. You're just working at a higher level of abstraction, which can be comforting, but simplifying would be to use the underlying systems directly or using a tool that only focuses on a single aspect of what you need (i.e. containerization).
> You don't need them. Everything from the post is defined in simple environment variables. For example socket activation is maybe 3 extra lines when done from scratch.
Great, then the article shouldn't import systemd bindings... My point is that the program is now tied to systemd systems. Containers don't impose such restrictions.
While this is a good writeup, and you end up with a service, you still need to manage a machine with all risks involved - server reboots, updates, networking etc.
AWS Fargate, or the new App Runner will manage a container almost hassle-free
It's not a criticism of docker. It's "docker does the same thing", because there are good reasons to abstract that functionality. I'm not familiar with podman internals.
> Why would I ever want to use containers with a tool that ...
Because you have to deal with those things always. You can do them yourself, you can do them with other services, or your can do them in the currently-most-common framework of systemd. You can still pick and choose from those elements as you want.
> You're just working at a higher level of abstraction, which can be comforting, but simplifying would be to use the underlying systems directly or using a tool that only focuses on a single aspect
I don't really see the namespaces and service management as different things. They're pretty much one idea these days.
It's also simplifying things because I can write my own script with the necessary unshare / mount / ip stuff. But the common patterns are repeating so much, I'd rather use a single option for it.
> Great, then the article shouldn't import systemd bindings
They're not bindings, just abstraction. Why should that be treated differently from "shouldn't import fmt, just concatenate strings", "shouldn't use rand, just open /dev/urandom"?
> My point is that the program is now tied to systemd systems.
Only in these examples. Actual apps normally autodetect the relevant variables and not rely on activation if it's not available. You're never tied to systemd activation / keep alive notification unless you choose to do it.
https://github.com/coreos/go-systemd/blob/main/activation/li...
I just used that rather than writing it myself because it felt like it didn't add much to the story. Also the code was actually written on macOS, it still runs fine without systemd.
Edit: I'd be quite interested to see if passing file descriptors in this way would be a goal in, for example, the proposed Alpine service manager. A quick scan of the proposal didn't show me anything obvious, but I'll go and read it in more detail.
As a programmer, I now don't need to care about dropping privileges, managing logging, daemonization (the moment I need to do the double-fork dance again, chairs will be flying, I swear), dropping into a chroot, and do half-arsed guesses "is it up and running yet?" from a convoluted mess of shell code that looks like a bunch of hair stuck down a drain for a month.
I just write an easy-to-debug program which I can launch from command line and see it run, and when I'm satisfied, a systemd unit from a cookie-cutter template is going to make it run. Service dependencies are now a breeze, too.
If I need to limit resources, I can just declare the limits in the unit. If I want a custom networking/mount namespace, it's taken care of.
I wrote about this previously here: https://kevincox.ca/2021/04/15/my-ideal-service/#socket-pass...
Or how about adding new buggy DNS code that doesn't work in common scenarios? Oh, sorry, here's another CVE because we didn't create enough test cases for the corner cases that are actually important.
Or oops, "nobody uses ntsysv", right? Or "We don't need to implement chkconfig even though we broke it".
systemd is a monolithic beast that is absorbing everything else in the system without considering that some of its decisions should be able to be disabled, and a lot of the design decisions are half baked. I don't believe in the philosophy of design its author has embraced. Progress is good, but please, stop breaking shit that has worked for decades. Anyone can write new code that partially implements a feature, but it takes real effort to responsibly migrate users from tools that worked to your new shiny half assed kitchen sink.
It may be great for less-skilled people, but for anyone running anything where it's too critical to outsource support it's then necessary to have a systemd expert inhouse (and such a person has proven extremely hard to find).
I can/could do without timesyncd and resolved (it's easy - just use chrony e.g.) but I like udevd being now part of systemd. It would be nice to not write /etc/udev/rules.d/ and instead have a foo.udev unit type, perhaps that is in our future (we do have .device units, but it's not the same - yet? the future). In this ballpark I think it's more on each distro picking and choosing - Ubuntu for example drank the kool-aide much deeper than RHEL - RHEL for example uses chrony out of the box, not timesyncd. However udevd and logind seem to be common across all distros now, as another user commented the KillBackground=yes setting in logind is just horrible to have as a default. The whole "homed" thing makes me sad that it's even being coded, I hope nobody adopts that (I dislike it for the same reason I dislike automount); someone out there wants it though.
The ability to dynamically edit a unit (systemctl edit) and to dynamically alter the running service constraints (systemctl set-property), all PID file type needs are handled in /run (getting rid of the nasty SysV stale unexpected crash reboot pid problem which many scripts failed to handle properly). Users having the ability to use their own private init items (systemctl --user) is great - timers, socket activation, custom login units, all very well extended down into the user's control to leverage. I'm sort of 50/50 on cron vs. timers, that's more of a use case by use case decision (example: tossing a https://healthchecks.io "&& curl ..." is just a lot quicker and easier in cron, but running a dyndns script on my laptop with a timer is nicer).
Touching on systemctl edit, it's really easy now to show folks (think a DBA team who only has the fundamental ops skill) how to quickly chain their After= and Before= needs for start/stop of their (whatever) without having to go down a rabbit hole - it's simple to use, the words and design are accessible and familiar, the method by which it works is a little obtuse (it's rooted in understanding the "dot-d" sub-include design pattern). On RHEL at least it uses nano as the default editor, annoying to me but good for casual non-vim users and easy enough to override using $EDITOR.
I used SysVinit for all the same years as everyone else (Solaris to Debian to Red Hat, ops touches it all) and wrote many my fair share of complex init units to start DB2, Oracle, java appservers (anyone remember ATG Dynamo?); systemd handles natively what 75% of that work was/is (managing PID files, watching/restarting failures, implementing namespaces/cgroups, handling dependency chains, etc.); for those complex scenarios (looking at you, Tomcat) you can still just have a unit launch a very complex shellscript "like in the old days". I haven't looked in awhile, but last time I knew in RHEL7, Red Hat did exactly that with Tomcat - just had the systemd unit launch a script.
It is, however, a real bear to debug sometimes - it's far easier to "bash -x /etc/init.d/..." and figure out what in the world is going wrong than it is to debug systemd unit failures. But, the same holds true for trying to debug DBus (if you've never tried / had to, it's not fun at all without deep dbus knowledge). I would like to see the future add more ops-oriented debugging methodology - if you've every used "pcs" (the commandline tooling for Pacemaker offered by RHEL), we could really use "systemctl debug-start" type of interfaces to the commandline offer the same experience as "bash -x" days of old. There are debug settings, they're just not ergonomically dialed in for the ops user, IMHO - systemctl debug-start would save people a lot of headaches.
This is, of course, not a problem, because as systemd folk are wont to point out, systemd is not, in fact, monolithic, meaning they use well-defined interfaces and can be swapped out for an alternative.
2. Some of the systemd logging is binary, so good luck with that if there's a problem.
3. Have you tried non-systemd init systems other than sysvinit?
4. Yes, it is convenient when everything below your development is cenrtalized by a single entity. It can easily provide a consistently useful underpinning. But there's a price - overly strong coupling of the init system, the kernel and part of the user-space, centralized control of, well, almost all of how things work on the system, and stagnation of the ecosystem due to there being only one game in town.
What other system is there, right now, that can do this so well?
I'm definitely onboard with the issues around tight coupling, I'm really not a fan of binary logs etc. But the unit files are pretty awesome IMHO.
So serious question - what else does those as well or better?
(Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
You can do the same painless setup (arguably even easier) with runit as the base template requirement is literally just
#!/bin/sh
the_executeable &
I remember when it took multiple days testing the configuration on different distributions, editions and versions to just get a single daemon process to start without failure. Then do the whole thing over again because the Debian based distros did not use the RedHat startup tools, different (or no) runlevel configurations, different service binders, different NTP daemon, different terminal deamon, etc.. And of course the French customers wanted the service to work with Mandriva, the German customers want SUSE support with extended security options like dropping permissions after boot.
Just like the article mentions you can define a portable service model with security and failure handling built in. There wasn't even anything that came close back in the day. Systemd may not have been built with the Unix philosophy in mind, but at some point that becomes a secondary concern.
Systemd unifies all systemd resources in units which work anywhere, its expandable and extendable, user friendly, allows for remote monitoring etc.
Like when logind was changed to kill background processes when you log out, by making KillUserProcesses=yes the default. Some Linux distros left that as is, others overrode it in /etc/systemd/logind.conf. So, figuring out what was happening, and how to fix it, was confusing. I had no idea it would have been logind doing that.
Similar for changes introduced with systemd-resolved.
And it leaves enough info (process is it's own separate session group) for logind to know it should leave it alone.
The ideas are not inherently bad. But they're not thought through, and the implementation is pure garbage.
Like taking the most stable software in the world[1], and going "nah, I'll just replace it with proof-of-concept code, leaving a TODO for error handling. It'll be fine.".
And then the "awesomeness" of placing configuration files whereever the fuck you want. Like /lib. Yes, "lib" sounds like "this is where you place settings file in Unix. At least there's no other central place to put settings".
[1] Yes, slight hyperbole. But I've not had Linux init crash since the mid-90s, pre-systemd
While systemd has a bunch of container-related functionality, it does not integrate well into the Kubernetes or even Docker workflow. It's used very little in those environments.
If you are building CoreOS or NixOS system images, or traditional Linux system services, then systemd matters. But I think way more services are being built for the container world where these problems are solved differently.
For example, the TLS configuration can be handled with common container patterns. The author's startup example would translate more easily to a full-blown Kubernetes environment once the VC funding hits their bank account if they had used containers from the start instead of first writing the service for systemd.
It's a shame because systemd is very powerful and I've enjoyed using it.
Putting a huge complex piece of software between yourself and "complexity" doesn't make the system less complex.
I sympathize with the "transition sucks" sentiments elsewhere on this post. Having a bunch of working scripts turned into instant technical debt cannot be pleasant.
But, as with python3, systemd seems to be the way things are headed.
One point is that processes other than root cannot start services on ports < 1024. That was a sensible precaution computers where big and multiuser, like in a university setting.
However, with single-serving services (e.g. in vm/container/vps/cloud), there is no need for it.
BSD lets you configure it with a sysctl option. But Linux defends that option like it is still 1990.
On NixOS, I patch it like this:
boot.kernelPatches = [ { name = "no-reserved-ports"; patch = path/to/no-reserved-ports.patch; } ];
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1331,7 +1331,7 @@
#define SOCK_DESTROY_TIME (10*HZ)
/* Sockets 0-1023 can't be bound to unless you ares uperuser */
-#define PROT_SOCK 1024
+#define PROT_SOCK 24
#define SHUTDOWN_MASK 3
#define RCV_SHUTDOWN 1You'll be defining your own systemd units with ease.
systemd to you will be journalctl and systemctl. So pretty good.
Systemd docs: https://www.freedesktop.org/software/systemd/man/index.html
List of directives: https://www.freedesktop.org/software/systemd/man/systemd.dir...
Unit-specific configuration: https://www.freedesktop.org/software/systemd/man/systemd.uni...
Service-specific configuration: https://www.freedesktop.org/software/systemd/man/systemd.ser...
Timer-specific configuration: https://www.freedesktop.org/software/systemd/man/systemd.tim...
The LoadCredentials thing reminds me of configmaps in K8S, is there a more general thing in systemd e.g LoadConfig
A more generic approach than LoadCredentials I think is the EnvironmentFile= directive if you want to pass along multiple env variables to your process without individual Environment= directives
It will be interesting to see if one day a replacement for systemd comes along and people who once championed systemd will begin to use the arguments the people who do not prefer systemd use to defend their choices for not wanting to use the next init system manager.
Part of the critique of systemd is the basic architectural choice of having this monolithic layer between regular user apps and the kernel. So, in a sense, the idea is _not_ to replace systemd with a better-written systemd, but to do things differently.
I might not be getting the point of the talk but I really appreciate the argument that Benno Rice presents.
Passing an arbitrary fd or socket from one process to another solves many problems and we are in the habit of doing it now.
It is probably unavoidable, looking at how complex modern compilers, processors and kernels are. They sure do make a lot of things simpler, though.
You will incur in lots of complexity trying to deal with init systems that aren't much better than traditional init.
Have you looked at s6? It’s a compelling alternative.
Churn isn't a virtue.
If there were any issues they would be worked on.
Things can be built to last, including software. That so many programmers today are not building such things (possibly they are incapabale) does not change fact that some did so in the past (whether intionally or not), and some still can.
Does everything you expect of single programmlet to manage your NTP, resolve.conf, DNS caching, mDNS, network devices, and etc.
Importantly, it weights only 1/100 of SystemD
Its not just breaking init.d scripts, it’s ntp, dns, syslog. Systems throughout the OS fail to things that were no longer short commands with muscle memeory, there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf
Even when you remember and type that in, you don’t get a simple list of nameserver and host, you get 100 lines of text you have to spend effort parsing to work out what’s going on.
When it’s less mental effort to run tcpdump port 53 to see where your DNS is going, there’s a problem.
For decades it was /etc/init.d/myserice restart
Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer.
Or the restart fails it doesn’t tell you why, it gives you two locations to look for log files about why it might have broken. Init.d scripts didn’t do that. Even if there was something really wrong that log files don’t reveal, running init.d with bash -x allowed easy debugging
Systemd came in and changed working processes and systems and gave very little benefit to people with working processes and systems from a operator point of view.
> there were now ridiculous convoluted commands like systemd-resolve --status instead of 30 years of typing cat /etc/resolv.conf
systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it!
Your rant does not sound very serious. Did you really have "ntp" or "syslog" in your muscle memory? That's strange, because most syslog daemon did not have a `syslog` command.
Anyway, systemd-resolved was created because it has uses. And for systems that used a dns cache (dnsmasq, etc), rejoice, because the config is now simpler than it was.
> For decades it was /etc/init.d/myserice restart
> Now is it systemctl restart myservice or systemctl myservice restart? I have no idea as I’m not at a computer.
Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`. Since sysv was unsuitable for many uses, several alternatives emerged, like "runit", or "upstart" for Ubuntu. So, before systemd, the recommended way changed with the distribution.
Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic.
> Or the restart fails it doesn’t tell you why, ... Init.d scripts didn’t do that.
In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure.
And `systemctl cat s1` display starting instructions that are rarely longer than a dozen of lines. I remember init.d scripts that were hundreds of lines long, and awfully hard to understand.
As a sysadmin, for me things like that were very, very minor issues.
The main problem was that systemd had awful documentation, written by people who'd clearly never had to use systemd in anger and just assumed that everything would work swimmingly (and please don't say read the man pages.. those are barely adequate).
When things broke there were no simple and obvious ways to fix it, you had to dive in to its labyrinthine spaghetti architecture and hope and prayed you somehow got the Rube Goldberg machine to work.
Hopefully that's improved by now, and there's some canonical documentation that really shows you how it all fits together and how to fix it when it falls apart.
* Services started with sysvinit would put logs where they want, which is fine if you know where they are but per-service you might be guessing. Having everything always in the same place is handy.
* sysvinit wasn't giving you any of these security benefits.
* If your system really was working fine before, why did you need to upgrade it to a newer distribution with systemd?
You should always use "service myservice start" instead of "/etc/init.d/myservice start". Running "/etc/init.d/myservice start" directly means the service ends up accidentally inheriting parts of your shell's state (environment variables, limits, current directory), which is a different environment from when the service is started at boot. The "service" command carefully cleans up most of the environment before running the script, making it much more similar to what will happen when it starts automatically on next boot.
And if you were used to "service myservice start", it now automatically forwards to "systemctl start myservice" when the /etc/init.d/myservice script does not exist, so it keeps working nearly the same after the transition to systemd.
Some things are a matter of preference, but this bit is just wrong. Init.d was hilariously worse. Some services had their own configuration locations, some had those exposed via /etc/defaults, some used syslog, some redirected stdout/stderr, some redirected one and discarded the other.
You're right that there are two places now - logs are either in the journal or in app-specific log location. And stdout/err go to the journal. Those 2 places mean fewer places to look through than we had before.
Thing is, it was never this command, it was always https://linux.die.net/man/8/service , but your command also worked in 99% of the times, until it didn't and restarted service misbehave. Systemd streamlined whole experience.
From what I remember, everyone else on UNIX world was already following up on systemd like systems before it got adopted on Linux world, and on Red-Hat/SuSE distributions daemon scripts existed for quite a long time as well, so no it wasn't * /etc/init.d/myserice restart* for decades.
Actually long before that it was service restart myservice, which still works.
For logging, can't you write a systemd service in bash with -x flag?
The best solution is to maintain infinite bash history and stop trying to remember arcane stuff.
Don't even get me started on tools like jq - so convoluted that I use python instead
I felt so alone in the world until this moment
However alpine is working on a similar thing based on s6 but with modularity and lightweightness as design goals. This sounds great to me. I'm not against the idea of a service manager, but I think systemd is overreaching.
I'm sure many people - especially developers who use OSX all the time - love systemd. That's fine, but people won't love systemd, just like solaris people learnt to accept linux.
I'm also sure not everyone loves it. Some have moved on, some haven't but now tolerate it, they've spent the time to cope with it, maybe it's costing them more time every day than pre-systemd, but it's not big enough problem to move on. That's just life. I'm sure some people didn't like it when program manager was replaced by the start button in NT 4 either.
It seems that systemd fanboys just can't acknowlege some people don't like their new world order, which is rather sad in itself.
What's missing from docs like [0] that makes you say that?
[0] https://www.freedesktop.org/software/systemd/man/systemd.uni...
Yeah, and it is not only underspecified, but too weak to be useful, which just pushes all the init.d logic somewhere else. What you've accomplished is moving it somewhere nonstandard, great job.
Also, the command line ergonomics suck. Systemd is deeply unfriendly to humans.
It was a power play in support of a long-term RH strategy, supported by a lot of bad faith arguments. Fascinating to watch as sociology, less nice as a forced-user.
What do you mean? Why the scare quotes? I'm not aware of anything in .service and related files which isn't declarative.
1. People to young to remember stable software. 2. People who have given up, and just accepted that Linux too "just needs a reboot everynow and then to kinda fix whatever got broken".
systemd has normalized the instability of shitty system software. And just like how you don't see front page news every day about 1.3M traffic deaths per year because it's not news, you don't see people up in arms about shitty Linux system software.
It's normal now. It didn't use to be.
Yes, ALSA is better than OSS, and then PulseAudio and now pipewire. It can do more. But when did it become acceptable to get shit, just because the shit could do more things?
Pipewire is not bug free (I have a bug that's preventing me from an important use case), but it's sure more reliable than PulseAudio, while still being more capable.
So maybe Pipewire is showing a trend towards coders actually giving a shit?
We still cannot block systemd from making a network socket connection so security model is shot right there by the virtue of systemd running as a root process.
In the old days of systemd, no network sockets were made.
Systemd has become a veritable octopus.
Now, I use openrc and am evaluating S6.
What do you mean by "security model" in this case? What model is that?
They might not have been better or more robust, but they where easier to understand and reason about. You could explain the entire thing to the most junior of sysadmins in a few minuets, tell them to read the boot scripts, and they would basically understand how everything worked.
And a few months ago I retried that with systemd. It's really just about 10 line of configs. And you are done.
Besides that, it also has a build-in scheduler with a command for you to tell when was the task runs, did it success? And what about the outputs. Although you could say it is just a cron replacement with better ui. But why no? I don't really care about the unix philosophy, I just care what do solve the problem for me.
Things would have been fine for a lot of people if they had stopped at SysV script replacing, and general start up.
At this point, with all the additional functionality continuously being added, I'm waiting for systemd to fulfil Zawinski's Law:
> Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
Systemd is 10 years in making, and still manages to brick production servers.
The problem is not with SystemD or its coding as such, but the ideology it came with, and bad developers who push it.
The last attempts to make it saner basically reverted it back to sysvinit. So, not much difference now.
Uhm, how so?
This provides a clean separation between the default configuration and the user configuration.
Can you explain why this is a bad thing?
A counter-example that comes to mind is when a package upgrade requires manual intervention due to file conflicts in /etc. That's what happens when the packager's default configuration interferes with the user's custom configuration.
> This provides a clean separation between the default configuration and the user configuration.
> Can you explain why this is a bad thing?
The normal way to do this is to put the default configuration in /usr/share/.
It's not. That's my point.
systemd does ask some good questions. E.g. I think the logfile situation needed a major shake-up in unix. Too many log file formats, in text, often completely unparsable (if you're lucky then a regex will work for 99.99% of log lines, but not all), and all unique. And the same mistakes being made over and over again. E.g. "oh, we don't log timezone", or even "meh, it's up to the user to parse the time with timezone correctly, even though things like 'MST' is not even unique".
But did systemd fix that? No. It's just that now I have logs in journalctl AND nginx, AND a bunch of other files. Thanks, standard number 15 that was supposed to unify it all. If you build it, they won't just come. Especially when the implementation is bad.
Believe it or not, the above is actually the pro-systemd argument.
Now, for what you describe: Yes. Exactly. I'm saying systemd DOESN'T do this. I'm saying this is a large successful part of Unix, that systemd ignored.
> A counter-example that comes to mind is when a package upgrade requires manual intervention due to file conflicts in /etc. That's what happens when the packager's default configuration interferes with the user's custom configuration.
Is that actually a problem? Maybe I've been spoiled by Debian, but with the combination of the upgrader showing the diff, and `foo.d` config directories, I've never had this problem in about 25 years of running Debian.
But I believe you if you say that others have this problem, and believe you when you say it's real. But how exactly is systemd fixing it? It doesn't resolve the conflict, if it works like you say where it's a per-file override. That sounds like breaking the system (usually in subtle ways) instead of hilighting the newly arrived inconsistency. That's just sweeping problems under the rug.
It's not enough to say vaguely that "this is different, so probably solves some problem, somehow. And it didn't cause me personally a problem, so fuck everyone else".
Perhaps blame lays on the distro packager, but still, it ends up to a user being strange.
I my experience, systemd config is simple because it handles all the complexity. Inside it’s guts, it is much more complicated than a sysv system — naturally so because it can do so much more. Those folks loves using the latest and greatest kernel function for all its glory.
All works well - until it don’t. When something is broken, suddenly you have to understand all the interdependent components to debug.
Back in the days, these were not so uncommon, because bugs or simply unimplemented features………
Here's an example: Someone read that fd-passing is a thing, so now systemd listens to just about everything and spawns stuff on-demand.
Now, that may seem like a good idea, if you think it up in a vaccuum and don't have experience with the real world. It's a great idea, if you're in high school. But to have it actually accepted? WTF is even happening?
Oh, let's do this for time sync from GPS. Great. All that time that could have been spent verifying the signal and all, completely wasted, because some jerk thought that it's better to waste 15 minutes of the human waiting, just to save 100kB of RAM.
It's a monumentally bad idea.
And more specifics: Like I said, when you replace init you need to have it not crash.
And then restarting daemons with systemctl almost to a rule fails, and fails silently. Often I have to just kill the daemon manually, and systemctl start it again.
But people aren't complaining about systemd anymore because now there's two kinds of people:
1. People to young to remember stable software.
2. People who have given up, and just accepted that Linux too "just needs a reboot every now and then to kinda fix whatever got broken".
But maybe the trend is changing? Pipewire looks like it's not actually shit (unlike PulseAudio which has plagued us forever), and while it has some important bugs in edge cases, it's actually more reliable than what it's replacing(!)
> As written, your statement is unlikely to convince anyone that isn’t yet already.
It's hard to convince people who don't care. Or indeed those who don't know that no, actually, short of a kernel upgrade "reboot to fix that problem whenever it happens" is not normal, and is a serious bug.
Pre-systemd Linux had as a selling point that it's actually stable, compared to Windows at least. But Windows has gotten much better in the past decade in reliability, and Linux much worse.
systemd is on the level of a re-think by a pretty bright high school student. And that's not a good thing. It's a very bad thing.
> to be convincing, it would have compare something like bug density to the software projects that collectively replaces
You're asking me to be data-driven, while being fully aware that systemd isn't, right? Your argument is essentially fallacy by implying that status quo is data-driven.
It's hard to take your suggestion at face value. Especially with many of the same people pushing systemd at the time making up shit like "We know that Unity is objectively the best user experience in the world"[1] (that's why it lost, because nobody liked it, right?[2]).
At the same time I also fall into group (2), above. I don't have time to wrestle in the mud with people who don't care.
[1] a quote like that, I may not have gotten the words just right. but the word "objectively" (without data) was there. [2] and I don't even particularly care about window managers. Before Unity I hadn't bothered switching from "whatever the default is on this system" in most cases.
Without (Docker) containers it is:
- build Go binary and install it in production server
- write and enable the systemd unit file
With (Docker) containers it is:
- write Dockerfile
- install Docker in production server
- build Docker image and deploy container in production server
I get the appealing of containers when one production server is used for multiple applications (e.g., you have a Golang app and a redis cache), but the example above I think containers a bit of an overkill.
* have a production outage because your libc was updated and now your go apps (which are dynlinked against it by default) won’t start
* mess around with low level cgroup settings if you need to oversubscribe safely
* cry in a corner the second you also need some python libs installed to do some machine learning or opencv or whatever on the side
Where you deploy your EAR/WAR file doesn't matter, so the application container can be running on Windows, any UNIX flavour or even bare metal, what matters is there is a JVM available in some way.
Also on the big boys UNIX club (Aix, HP-UX, Solaris,...) systemd like alternatives have been adopted before there was such an outcry on GNU/Linux world.
On cloud platforms if you are using a managed language, this now goes beyond what Java allowed.
You couple your application with a bunch of deployment configuration scripts, and it is done, regardless of how it gets executed in the end.
The cloud is my OS.
That's another problem to be solved.
There is no need for PrivateTmp= or some of the other configuration shown in this article because the application container already runs in a separate environment.
I think this is worth considering with respect to this article, even though containers definitely bring their own problems.
I use NixOS but certainly not love systemd. Instead, I've created a way to replace it with s6.[1]
1: https://sr.ht/~guido/nixos-init-freedom/
Cheers, Guido.
How does NixOS make that easier?
For example, systemd by default permanently gives up restarting services after a few number of tries (e.g. 5), even if you have set `Restart=always`. This is suboptimal for web servers that should recover by themselves after arbitrarily long failures (e.g. network downtimes outside of your control).
On NixOS, you can, from your machine config, set:
systemd.services.nginx.unitConfig.StartLimitIntervalSec = 0;
We use NixOS for our infra (having used Ansible before), and this ability to override anything cleanly and keeping defaults otherwise made for much easier to maintain infra code and less ugly/surprising compromises.
when you have to run "systemctl disable systemd-timesyncd systemd-resolved systemd-networkd" to start to get back to sanity, that's not init
If you can bind port 80,you can gets ssl certs via let's encrypt (which could let you intercept not just web, but also smtp/imap etc).
So yes, it can make a difference. Of course - it's better if the user doesn't have access to begin with.
This might be more interesting for classical multi-user servers than "single use" servers that don't allow "regular" users to login via ssh.
If you only mean to say that you replace 3 bits of systemd then it would be good to say that, and to say which bits of systemd it replaces.
A lot of distros do not even use those systemd services, for example ubuntu does not use systemd-networkd by default.
I read "it weights only 1/100 of SystemD" as if it compared systemd to connman directly.
https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...
I feel your pain - I'm curious, was it a drop-in /etc/logrotate.d/ config which was sending a HUP? I don't run squid but Google'd up that it will take a USR1 to rotate logs which should not close the HTTP connections (allegedly HUP closes them). Perhaps the AL2 folks chose the wrong signal? https://wiki.squid-cache.org/SquidFaq/InstallingSquid#squid_...
When the systemd people say session, they mean you would have to specifically do some kind of dbus call I'm not familiar with.
Typically this was an issue on B) workstations, and B) servers, where you might log in as a regular user without super user access.
And this[1] (might?) now work as advertised in the documentation - but the discussion around the bug doesn't induce great confidence...
https://github.com/systemd/systemd/issues/3388
[1] example no 5 here: https://www.man7.org/linux/man-pages/man1/systemd-run.1.html...
> They can easily register a user service and continue running there.
So systemd came in and broke the 50-year-old way of doing things, and now the programs that got broken should all have to add systemd-specific code to work again?
Yes. Just because it is 50-years-old doesn’t make it suddenly a valid approach. There are 4 states here - process running with usual semantics for closing, process running that would like to linger in the background, and the frozen version of this two. How should a service manager/resource handle — that is very much in-scope for systemd differentiate between a frozen process that should very much be cleaned up in the no-linger case, and any process wanting to linger? And there are well-established ways to determine whether a process is frozen — pinging it expecting a specific answer.
Systemd differentiates between no-linger/linger by introducing user services - one should only write a trivial alias/wrapper script for tmux and can continue to use it to their liking. Also, it has a compile time flag, as well as a runtime config one - it is up to the distro or the user to revert back to the old way.
service myserver restart
/etc/init.d/myserver restart
$ service myserver status
$ service myserver reload
$ service myserver restart
# /etc/init.d/myserver reload
unknown command reload
# /etc/init.d/myserver restart
unknown command restart
# /etc/init.d/myserver stop
myserver: warning: frobulator did not fully unfurl
# /etc/init.d/myserver start
myserver: still running on pid 1234
# kill -TERM 1234
kill: (1234) - No such process
# grep "running on pid" /etc/init.d/myserver
echo "$0: still running on pid `cat /var/run/myserver.pid`"
# rm -f /var/run/myserver.pid
# /etc/init.d/myserver start
My use case is to get my VPS run a web and mail service on Nixos but without the bloated binary logging of journalcontrol. The indexes in these log files change heavily between snapshots so they take up way more disk space than append-only text logs, that snapshots very well on ZFS.
My experiences while building:
- It's easy to use the config.system.services tree of the user-services (ssh, bind, caddy, etc) to create s6-services;
- Sometimes it needs a change to make it work [1] on s6;
- That same change seems oblivious to systemd. I guess it just starts the process and never bothers to monitor liveness. So much for a process management system ;-)
- Nixos packagers seems overworked, as my pull requests seem to get stuck ;-(
- Nixos use of systemd leaves a lot of decisions to resolve at boot time, decisions that I want to make at build time with s6;
- However, I cannot create a s6 dependency tree specification at build time, that's still at run time;
- Because S6 uses the service-directory to store state-files in the same directory (no /etc-/var split).
I can pattern match that https://blog.darknedgy.net/technology/2015/10/11/0/ - "Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise" from six years ago (Discussion at https://news.ycombinator.com/item?id=10370348 ) might be relevant.
Are those points reasonable? Dunno. If so, have they been addressed? No clue.
Because the only two alternatives are 1) running mainstream distros from before 2014 or 2) running obscure distros that still don't use systemd.
Contrary to “popular misconception”, Linux is not a settler's freehold where every holdout makes their own rules.
Admins have to work with a diversity of systems. They can choose how to setup new ones, but they have to work with a range of them. What everyone does has an effect on everyone else. If some distro introduces a new way of doing things, it has some chance of ending up affecting a lot of people - they might have to adapt to also support that way, or handle it in some way.
In that way we are a village, things are connected, and that’s why there is some degree of "social control" - looking across the neighbor’s fence and meddling with their way of solving the problem - it can in some sense become ours, if we are unlucky.
Fortunately, we can relentlessly copy good solutions from others in the village too.
You can contribute to your own distro, and the "veteran UNIX admins" made Devuan. If it still counts as obscure and you don't want it to be, you - yes, you - can do something about it.
The free/open-source community is a do-ocracy. The things that are worked on are the things the people doing the work want to work on. If you have a well-paid sysadmin job where you are providing your employer value by using the work they provide you for free - and, in particular, using their ongoing work which they continuously provide you for free, because you feel like a mainstream distro from 2014 doesn't suit your needs - then you can either be grateful for what you get for free or you can contribute back.
(Which doesn't necessarily have to be contributing your own work. I'm sure if you get your employer to donate one FTE's salary to Devuan, you can change its obscurity pretty quickly!)
Traditionally developers would write code and packagers would package them into a distro specific rpm/deb -- including the init scripts (which may be pushed back upstream). Developers wanted to bypass this slow process, and there were far more developers than people willing to package the software.
Personally I've never had a problem writing an init.d script.
systemctl | grep ssh
systemctl cat sshd
I find it useful to find the unit name, and 'cat' displays the location.
I’m not so sure there isn’t any room for improvement though. In the related s6 project there is lot of discussions about adding new features that are beneficial for supporting a modern Linux distro.
I haven't once been in a situation where I thought systemd would help. Granted, I don't run many custom services, but for what it does, runit does a great job. Something a bit more user friendly like s6 would be nice, but otherwise it stays out of my way and I don't think about it.
NixOS/Guix would be worthwhile switching to, but Void is comfortably simple.
I think most of the resentment against systemd is that it felt like it was forced down our throats. Systemd may be an improvement over the mess of shell scripts we had before but it isn't perfect. It's one thing to choose to give up decades of experience so one can voluntarily switch to a new system. It feels very difference when the choice isn't one's own.
Part One:
---------
> systemd-resolved is not enabled by default in Debian and many distributions, and it is not needed in any way by systemd. If you don't like it, don't use it!
It's not possible to ask systemd about which parts are enabled and up to which extent. It always adds a discovery phase before starting to make changes in a system. If you don't do this discovery, you're probably in a wrestling party with systemd. If you do this discovery, it costs you time. Systemd SHALL provide a way to see how much of its enabled up to what extent.
Network Manager got this right. If there's a distribution native configuration file for an interface, Network Manager ducks out and leaves system to its own. If systemd finds an equivalent service running both in its own ecosystem and from another package, it either overrides it or collides head-on with it, so either systemd equivalent works, or nothing works as expected. Neat(!).
> Before systemd, at least on Debian, for a few years the recommended way was NOT calling `/etc/init.d/something`, but instead `service apache restart`.
That'd be RedHat family of distributions. Debian doesn't have service command out of the box. Using both for over a decade, RH uses service, Debian uses /etc/init.d
> Thanks to systemd, most linux installs now use `systemctl restart service1 service2`. Note that you can now act on multiple services at the same time. You can use this feature as a mnemonic
There was a thousand ways to do that before systemd, systemd added yet another way. It's not bad, but it was not novel in any way.
> In many cases, init.d scripts told you nothing when they failed. Each service has its own procedure. Nowdays you can always see what happened with the command systemctl prints on failure.
systemd reports the failure in the lines of "service failed to start. go look to the logs. I also probably intercepted them, so journalctl may work".
Any good init script reports the error as "I cannot find my run file and/or process, please go get the logs. Something is probably borked". It's more/less the same thing. Better reporting in systemd requires systemd targeted service files, which are not mandatory, or most practical all the time.
Part 2:
-------
Whenever something comes up about systemd, this summarized conversation comes to life:
U1: We were happy before systemd. It broke too many things, it also tried to hijack everything. Now everything is different.
U2: *defend systemd in various and countless ways*
U3/U1: You're wrong.
*A flame war ensues for some time*
However, neither of these systems have this fierce defendant army of systemd. To be brutally honest, systemd has a lot of advantages, makes many things more practical, and faster in many ways.
OTOH, systemd is not radically fast. It's faster, but not blindingly. It's practical, but not always. It's hijacking of services makes some stuff very backwards. Replacement of NTP and resolved makes things hard to manage. Default binary logs makes some admins and external systems go blind.
I'm not against progress or systemd in general, but please be a little more accommodating for neutral and negative comments about systemd. Not all the commenters are bone headed caveman who love their flint stones and reject lighters with all their life!
It absolutely is possible to do this. It's the same as finding out if any service is enabled or not. Every view of a service includes it's state and whether or not it is enabled. So if you want to check an individual service, look at systemctl status $service, and if you want to look at all services: systemctl list-unit-files
And if you want to look at only the enabled ones, you do the Unix thing and "| grep enabled"
I don't see why systemd needs to hardcode a command for this.
But this comment is totally right. If you're a Linux sysadmin, how hard is it seriously to type into a search engine "systemd list enabled services." Exactly this command very helpfully comes up in DuckDuckGo's knowledge graph bubble so you don't even need to follow the link to askubuntu. I'm sure Google search does the same.
The difference here is that systemd actually went to the individual distro maintainers and listened to their concerns, made the necessary changes, and convinced them all to adopt it. That's damn hard to do in the Linux world, I commend anyone who can do it successfully.
Regarding your part 2: For whatever reason, there is an absurd amount of misinformation posted whenever systemd comes up. If you posted something that was wrong about some other service manager, I would correct that too. You deserve to know the right answer to things, for your sake, not for the sake of systemd (or any other program). Please don't dismiss attempts to correct misinformation as being unproductive, it's the flame war which is the unproductive part.
Why I need NixModules in the first place? ELI5, please.
E.g. you could write a NixOS module to manage your web app. You declaratively configure that you want to run Nginx, Postgres, etc. open some ports, connect to your VPN and basically everything else you might want. There are a lot of existing modules, so in most cases you have to write very little code yourself. If you want to scale your system to run on multiple systems or containers, you can do that with relative ease.
You also have a singular source of truth for all of your information, down to your application binaries. Your configuration says you are using Postgres version X.Y.Z with compilation options A, B and C, so that's exactly what is running on your systems.
In 10 years, though, reading recent bug reports, I can see that the project's leads have grown out of it, largely.
Case in point, they had recently put a bug into a 24x release that made a lot of machines, including mine, unbootable, but the fix was just as quick.
We had 3 major breakages due to systemd, which is why we went back to something pretty much anyone who knows a bit of bash scripting can fix.
Yet another person writing fantasy, not facts. Systemd wasn't "quickly" made a hard dependency. It was a soft dependency for ages, then eventually the release team made a mistake around systemd-logind. I was part of the GNOME release team at that time. Still, GNOME runs without systemd. Meanwhile we had loads of discussions with loads of distributions.
Yet the things you write: quickly done, apparently lots of people were secretly paid by Red Hat, forcing distributions? All devoid of any facts, just emotions that dismiss the amount of work me and loads of volunteers have done.
Also if intransigent network connection occurs (fuZzing or not), untested errors occur … notably in PID 1 which isn’t easy to debug.
And it’s codebase is too ginormously huge.
Why does this "not compose well" ?
You don't have to override the whole unit as /etc/systemd/system/nginx.service , which would have problems if two things wanted to override different parts of the original unit. Just drop an override file in /etc/systemd/system/nginx.service.d/90-restart-always.conf with that one specific config you want to override.
Because you cannot easily write libraries/components that do this.
In NixOS, other modules can override the options of other modules. For example, a a web app can set the nginx options that it needs, instead of requiring you (the admin) to "drop a file" in /etc.
This is one of the reasons why on Ansible Galaxy (community repository of Ansible roles) there are 527 nginx roles [1], and in NixOS there is 1 nginx module that everybody code-reuses.
[1]: https://galaxy.ansible.com/search?deprecated=false&tags=web&...
systemd.services.nginx.unitConfig.StartLimitIntervalSec = 0;
If the point is that it's not manipulating a system-level nginx service but a user-level one, then writing the systemd override file in the way I described doesn't require root either.
If you want to claim that process was democratic, you have to believe that the members of the committee perfectly represent the opinions of all Debian developers (to say nothing of Debian users), and therefore have to excuse the fact that the vote had to be settled by giving one of those people, Bdale Garbee (HP's CTO of Linux), effectively two votes.
It was only three days after the tech committee's decision that Mark Shuttleworth announced that Ubuntu would fall in line by abandoning Upstart[0], and not until many months later that the first General Resolution was put forward to try overturning the committee.[1] Of course by that time everyone was tired of the arguments and it would have soured relations with Canonical to force Ubuntu back away from systemd, so the GR was doomed from the start.
[0] https://www.admin-magazine.com/News/Ubuntu-Abandons-Upstart
But it's also overengineered. Like starting a daemon on first connect is "neat trick", but should never have gone beyond that.
Like: oh, I want to restart this daemon, because I want it to re-run its init code (possibly with new settings), but you CAN'T, because some idiot decided that it'll only actually start when someone connects to its unix socket, so running "restart" is a no-op.
FTA:
> It can then boot your service on the first request, or you can do systemctl start lunchd yourself if you think that would take a while.
Sounds like you're arguing something like "yes, it's broken. But you can just reimplement everything and it won't be broken anymore".
Yes, I can also turn off "kill user processes on logout", but that doesn't make that not-broken.
Like, who even boots their computer anymore? Isn't the typical user on a laptop, and just suspends it?
My workplace even had to install corp software that forces a reboot every N days (with warnings ahead of time) because people just Do. Not. Reboot.
And even for the people that do, at what cost, here? You have a bunch of services and services completely broken, but "they started just fine" (except they didn't start), and only break once you actually need them.
So to me this really looks like it applies neither to servers nor desktop. I'm really not seeing any use case except fetishizing boot times.
And for me this always SPENDS human wait time, not save it. I try to use a service, and nope, it needs to "boot up" first. Could you not have done that already, WTF? (and maybe it fails to boot, which I only find out about now that I'm already in the zone to use it)
Are we really optimizing for kernel developers, here? Can't they just disable the services they don't need, to speed it up?
And we have eleventy billion cores now. Really? You can't start a 645kB gpsd? It takes what, 3ms?
(My gripe with how systemd does a traditional init’s job is mostly the unit and dependency types, all of which AFAICT are specified in terms of actions on state transitions, not consistency conditions on states, so for all that it has a positive boatload of them I can’t actually figure out how to specify which configurations are permissible for my system.)
Unix has always been worse is better. You may disagree but it is the secret sauce. YAGNI for that potentially ideal system. Sadly. I like mathematically solid systems but the people with good enough systems explore the solution space much quicker.
Indeed I don’t even necessarily agree with everything said on the page I linked to (which I did reread before posting the comment), but regardless of what I think about the particular point in design space (s6) it advocates, I still consider it a good overview of the space itself and prior art in general, and that’s the only thing I claimed to offer. I do have some thoughts about init systems, but I don’t feel they’re ready to put them up for discussion here, so I haven’t.
Where systemd has one file with a bunch of settings in, this is split into a directory of single-purpose files in s6. I'd hesitate to call it "better", but from a surface reading it seems roughly equivalent from a usability perspective.
"Debian GNOME packagers are planning the same AFAIK; they rather just rely on systemd (as init system, not just some dependencies). In the end, the number of distributions not having systemd decreases."
That was written by Olav Vitters, of the GNOME Release Team, who later admitted[2] "Personally I’m totally biased and think the only realistic choice is systemd."
You could argue that any blame for this dependency therefore lies with the Debian packagers, rather than Red Hat employees, but actually, if you look into the history, there was already a push for making GNOME dependent on systemd three years earlier by none other than Lennart Poettering.[3]
Even at that time, Josselin Mouette, founder of the Debian GNOME team, obsequiously replied "I don’t have anything against requiring systemd, since it is definitely the best init system out there currently" and later acknowledged the influence Red Hat had over the direction of GNOME, saying "Red Hat being the company spending the most on GNOME, it is obvious that their employees work on making things work for their distribution" and "on the whole we don’t intend to diverge from the upstream design, on which a lot of good work has been done."[4]
So there was definitely pressure on Debian from GNOME to make systemd the default init, and pressure from Red Hat to make GNOME depend on systemd. Whether or not these decisions were all coordinated in advance in a smoke-filled room is beside the point, given that things worked out exactly the way such a conspiracy would have wanted.
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=727708#5
[1] https://blogs.gnome.org/ovitters/2013/09/25/gnome-and-logind...
[2] https://blogs.gnome.org/ovitters/2014/02/03/my-thoughts-on-t...
[3] https://mail.gnome.org/archives/desktop-devel-list/2011-May/...
[4] https://raphaelhertzog.com/2012/01/27/people-behind-debian-j...
Much like Pipewire exposes PulseAudio interfaces but is implemented differently.
People spend a couple of years getting used to a stack in their early carrier and then spend decades arguing that it should never change.
A lot of that "python2 will never die" crowd left python all together, and they are better off for it because they won't have to deal with the next time python decides to throw everyone's work out the window.
Even when we can show am improvement in security and usability, and lower training cost because of consistency, it's still another mouth to feed.
Why are you talking in the past tense? We have done no such thing. Death to python 3; long live python.
But you are right, adoption is not enthusiastic, which to me is a massive indictment of the design and usability. We'll basically spin wheels until someone gets annoyed with it and does systemd better, or at least more modularized.
My complaint? sudo systemctl <verb> <service> means the verb cannot be autocompleted or introspected like sudo service <service> <verb> could be. May be minor, but it's generally my only interaction with systemd versus init.d, and to me they completely blew the only thing I use. Not a good impression.
I understand that init.d was a cobbled set of scripts, loose conventions, and even some hacks. But the resistance to system.d is so pervasive it cannot just be stubborn unix neckbeards.
Why else do you think so many distributions switched?
Yes, the resistance is noisy and stubborn Unix neckbeards. Not even Unix, since every other Unix had something similar to systemd already. LINUX neckbeards.
>systemd is on the level of a re-think by a pretty bright high school student.
That seems like an odd statement, I believe systemd was inspired by other established unix service managers like macOS launchd and solaris SMF. The design is definitely not perfect but I wouldn't say the history was ignored when making it.
First of all this just sounds like "I don't know what your problem with systemd is, you can just choose to not use it". Second, my point is that it's an absolutely terrible idea, and should never have been done.
> I believe systemd was inspired by other established unix service managers like macOS launchd and solaris SMF. The design is definitely not perfect but I wouldn't say the history was ignored when making it.
A high school student can read up on things and then make something they, without real world experience, thinks seems like it's a good idea. And with no experience about what it takes to make software reliable.
If you have experience making software reliable, please consider submitting bug reports and patches to help the project, like you would do with any other open source that you depend on. I'm sure contributions to improve the testing and CI would be appreciated. A high school student can also trash talk loudly about things they didn't take the time to fully understand (and I admit I did a lot of that when I was a teenager in high school), but it takes real expertise to illustrate what the actual problem is and to contribute a fix for it in a positive way.
This feature benefits some people. Even if your service requires socket activation, you can still set the service to startup and you can still control it with normal command lines.
> It's a great idea, if you're in high school.
You make this accusation twice, but you give no-one any reason to believe it (unless you think "using an optional feature I don't want to use and which isn't significantly easier to use than to not use" is a reason, but most people would say that it is a good example of flexibility).
> And then restarting daemons with systemctl almost to a rule fails, and fails silently.
I have encountered an issues with using `service blah restart` on Ubuntu, which means if it isn't connected to an interactive session it doesn't work properly. I was able to fix this by switching to `systemctl restart blah`. Perhaps you're experiencing something similar? I imagine Ubuntu's service wrapper is probably taken from Debian so it could be quite widespread.
The fact that `systemctl (re)start` doesn't always give useful feedback is irritating. I am usually too bothered by something not working to have noticed when and why it gives no output about a failed service. A command should always output something on error and it should be sensitive enough to notice whether it has succeeded or failed.
> Linux too "just needs a reboot every now and then to kinda fix whatever got broken".
This is another accusation you make twice. I have been using Linux since the 90s, and I cannot agree that we have to restart Linux distros more often now than before. Can you give any examples of circumstances when you choose to reboot?
> You're asking me to be data-driven, while being fully aware that systemd isn't, right? Your argument is essentially fallacy by implying that status quo is data-driven.
Systemd might have been adopted on theoretical grounds. But that doesn't mean that an empirical objection is useless or irrelevant. If you can show that the theory doesn't match the data, or the data is worse for systemd than some alternatives due to unforeseen consequences or the difficulty of dealing with the larger spec, then this might lead to improvements to systemd or adoption of some alternative.
> At the same time I also fall into group (2), above. I don't have time to wrestle in the mud with people who don't care.
You don't fall into group (2) above. Group (2) is a subset of people who do not complain about systemd, but you are complaining about systemd. Moreover, your comment slings a good lot of mud, so it's hard to take that as a valid objection. You at least should work to clean up the mud you threw unnecessarily.
Yeah from the skarnet page above. BIND had many troubles over the years but mostly because we were all learning about secure code practices in C etc. bind was just a name server. Never heard of it reading mail. There was a lot to learn about network exploits, for sure. But attacking bind as doing too much seems disingeneous. And the whole tone reminds me of the same tone as https://suckless.org/ who cares if some browser takes a Gig of RAM or my window manager is ginormous. My laptop is hard pressed to use all the RAM it has and if systemd and the kernel is running sixty daemons but the unit files are easy to write as a here doc in cloud init, then win win win.
(Running postfix and bind on my personal cloud VMs; have run Apache, haproxy, nginx and lighttpd, as well as built in Python web servers).
But it's broken by design. It's a bad idea. It's "neat", but "neat" doesn't add value.
> You make this accusation twice, but you give no-one any reason to believe it (unless you think "using an optional feature I don't want to use and which isn't significantly easier to use than to not use" is a reason, but most people would say that it is a good example of flexibility).
Not sure what you mean. Using this feature pushes orders of magnitude of complexity onto users, and greatly reduces the ability to error handle or even know the status of services.
Having a daemon listening to a socket is clearly orders of magnitude easier for end users. It means everything is in agreement. You check if the service is turned on (systemctl or whatever), you check if the process is running (ps, etc), and you check listening ports (netstat, nmap, etc...), they all agree that the service either is or isn't running. And if it's running, it's successfully run its initialization and should be usable.
We already have this experience with inetd-based services. fd-passing is reinventing the past, poorly. fd-passing actually would have made more sense back in the 90s, because spawning a new process was more expensive. Both in terms of code and CPU power that overhead doesn't actually matter anymore for the vast majority of cases.
Again, I don't know what your question is. Is it "why would I possibly want to know if a service is ready do its duties or not?"? Because if it is, I don't know what to tell you.
> Perhaps you're experiencing something similar?
I mean things like restarting nginx, and either it just plain didn't (and thus fresh TLS certificates weren't picked up), or it failed to start up again and now nobody's listening to port 80/443 at all.
> Can you give any examples of circumstances when you choose to reboot?
I've filed bugs, but don't want to doxx myself. Something more systematic is that sometimes after running apt-get upgrade it's recommended that I reboot (for non-kernel reasons). The fact that someone would even write that message is a sign that the author doesn't care.
> Systemd might have been adopted on theoretical grounds. But that doesn't mean that an empirical objection is useless or irrelevant.
I agree. But this is a very common tactic for people who just don't want to have a discussion, too. I'm sure in this case you're saying it in good faith, but you should be aware of the assymetry of asking one side to provide data when the other side has none. And the cost of collecting interpreting that data (depending on which parts of this, what, a human-year?), and the risk of systemd people dismissing that data anyway, because "yeah, I guess the data supports your point of view, but I don't like it so you can fork the repo to implement it if you want. Bye.".
So I'm not saying this appeal to data is in bad faith, but it is a bit naive.
> Group (2) is a subset of people who do not complain about systemd, but you are complaining about systemd. Moreover, your comment slings a good lot of mud, so it's hard to take that as a valid objection. You at least should work to clean up the mud you threw unnecessarily.
Best comparison is that I can complain about the corruption of politicians without inviting an argument that I myself should become one, to drain the swamp.
IOW: I don't have the time, and if nobody else cares, then even if I did then I don't see that I would succeed in a sea of people who don't care about software reliability.
You keep saying this, without giving evidence to back it up. People are still running Linux in memory constrained environments, those didn't go away now that the 90s are over.
>Using this feature pushes orders of magnitude of complexity onto users, and greatly reduces the ability to error handle or even know the status of services.
To be clear, it sounds like what you're suggesting is that these services implement their own fd holding logic, which is going to be even more complex, and is exactly what systemd is trying to prevent from happening.
>You check if the service is turned on (systemctl or whatever), you check if the process is running (ps, etc), and you check listening ports (netstat, nmap, etc...), they all agree that the service either is or isn't running. And if it's running, it's successfully run its initialization and should be usable.
This isn't really correct, netstat or nmap won't show process status at all. You really don't know what the real status of that port is unless you've run lsof or something else that scans the open fds of the processes, and such a tool would make it obvious when systemd (or some other fd holder) has the fd open. Also, systemctl will display this separate socket/service units, so you can just check if the socket unit is running but not the service.
What I mean is that services can set other services' options, without you (the admin) having to to write such overrides manually.
I guess it could be preferred as a second-order factor.
You've rewritten history to pretend you're right. Then you follow up with more drivel? Sorry, aside from trolling, what is your point?
This seems pretty clear that GNOME used to depend on systemd. How am I rewriting anything? https://wiki.gentoo.org/wiki/Hard_dependencies_on_systemd#Pa...
IMO Lennart is right in that collection should be optimized to a point where it is (almost) never the bottleneck. I, personally, would put a filtering program between the chatty application and journald.
- Legacy application, with no access to source code.
- Company that wrote the application hasn't existed for a decade.
- It logs several useless lines every second. This is non-configurable, and includes a timestamp which makes each linue distinct.
- It also logs some very important lines that _must_ be responded to.
I don't think this is a unique situation to find yourself in, and currently this application is permitted - explicitly - to spam the journal to a point that it is effectively useless. Yes, you can use a filtering application to sit in between systemd but that's a workaround and not an actual solution (as stated in the issue linked above):
- it could not be "plugged in" via overlay (i.e. modification of Exec... is needed - this gets ugly especially if you have ExecStartPre/ExecStopPost)
- SyslogIdentifier has to be set to produce proper name in journald
- The filter itself has to be quite reliable (to not hang) and a bit sophisticated to properly handle signals, process shutdown, low disk space etc.
Rate limiting works, but it’s a pretty blunt instrument and can lead to losing your actually valuable entries vs capturing garbage.
But this is really a moot point. Systemd's socket activation is really meant for system services which would otherwise be in the critical path of system boot. 'Regular' client-facing services that people normally run–webapps, etc.–are not really the target use case. It's fine to start them up in the normal way, with WantedBy=multi-user.target in the [Install] section. And I have never seen people use socket activation for them anyway. So you are basically arguing a strawman here.
gpsd is an example that immediately comes to mind that was set to start on-demand. Which is ridiculous.
So it's a straw man that actually exists, making it not a straw man.
For example, if the system clock is not monotonic, my text logs are still written sequentially in order, and it's easy to figure out what happens. It is the stated view of the journald maintainers that, if your clock is not monotonic, you get to keep both pieces. https://github.com/systemd/systemd/issues/662 (The specific issue described there sounds complicated, but the fundamental problem behind it, which I have hit multiple times, is "if your system can't maintain a monotonic clock at all times, including early boot, then your logs will be mangled.")
Similarly, a text log line is complete the moment it's written, even if the process that wrote it goes away immediately afterwards. Journald, by contrast, performs asynchronous metadata lookups for _each_ log line at some later time, which it apparently does not cache. This means that, when a process dies suddenly, the final messages will not appear in "journalctl -u", because they failed to get tagged with metadata. (I can't find the bug for this, so I haven't verified that it's still open and unfixed at this time. But it has certainly existed for years at this point.)
So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
https://github.com/systemd/systemd/issues/2913
I am so happy you mentioned this, as I am in a position of developing a daemon used by people who insist on using systemd, and I keep asking them for logs when things fail and, in fact, the most critical logs -- arguably the only ones that ever truly matter: the ones that come immediately before the daemon terminates for some reason -- are often missing. Now I know why :/.
https://unix.stackexchange.com/questions/291483/how-to-fix-u...
> So no, in principle I have no objection to _competent_ binary logs. Journald does not meet that bar.
Yeah... I hadn't even gotten to the end of your message before immediately going "omg I have to look into this" and the most apt description that was coming to my mind for a logging system that fails to actually store logs -- and particularly the most critical death throws to the log before termination -- is 100% "incompetent" :/.
It's subtle, and you've accidentally missed it like many do - you made an assumption that everything logs through journald. On my personal system where I do not have rsyslog (aka trying to live the journald life) I have non-journald text logging for: httpd, sa (sar/sysstat), lightdm, audit, atop, Xorg, cups, fdsync and samba. If we just stick to httpd, sa and samba (most folks know how those work) it shows how logging is way more complex than what is captured in the journal - these apps by design maintain their own logs.
So now you launch your rescue ISO (let's assume modern sysrescuecd which has journalctl) and get your filesystems mounted, you have to employ two different techniques - journalctl for that single-use format and then your traditional find/grep skills for everything else. You don't know why it crashed, how it crashed and are on a fishing expedition. Was it an RPM upgrade? (logged to yum.log or rpm.log, not journald) Was it update-initramfs running out of disk space truncating your initrd? (sometimes logged to journald depending on distro, sometimes not). Don't know until you start following breadcrumbs, find/grep is the superior toolset (much like the trebuchet is the superior siege weapon).
It's not that you cannot do it, it's that journald-only (no rsyslog) forces using a specific method with specific tools to access what should be extremely easy to access data. Keeping logging all in the same format (text) is what I want, it's not that I don't know how to use journalctl; I don't want to use journalctl, it's a pig in lipstick.
> (Not to mention UNIX has log files that have been in a binary format since time immemorial, like utmp and wtmp.)
I agree with this, they belong over in /var/lib/ somewhere and those files bug me, always have. "Just because these other guys did it" is however a logical fallacy, they are (IMO) just as wrong because they're more like database files than they are logs in my opinion. (the secure/auth.log is more a "log")
In theory it is possible to make a binary log database which will work fast for many queries including these, but journalctl is not fast at all.
2. People are fine with gzip because log compression is usually done in a way which makes log corruption/loss highly unlikely:
gzip log.ystd, fsync log.ystd.gz, rm log.ystd
At any time you can reset a server and you will either have uncompressed or compressed logs, not some partially written binary blob.
Or if you're writing to network storage and would like to analyze the logs from your haiku box.
Text is not perfect, but it's the one thing that is always available.
Also I'm not sure about the journalctl format, but in general binary formats don't handle partial corruption well. Which is something kind of important for logs.
It's not a problem - it's just not UNIX.
"Give me an IP, username and password - what OS is it?" are about all you start with and go from there. It's probably a critical system to someone, and everyone swears on a stack of bibles that nobody did anything, touched anything or made a change. You have very specific domain knowledge (kernel, grub, SAN/storage, systemd, dbus, etc.) and typically ask a lot of questions to the systems owner as your fingers are flying ruling out reasons (low hanging fruit common issues).
Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT. It is encoded in 7-bit ASCII with records delimited by 0x0a bytes.
Corruption tends to be missing data, and so the reader has to jump ahead to find the next synchronization byte, aka 0x0a. This also leads to log parsers producing complete trash as they try to parse a line that has a new timestamp right in the middle of it.
Or there's a 4K block containing some text and then padded to the end with 0x00 bytes. And then the log continues adding more after reboot. Again, that's fixed by ignoring data until the next non-zero byte and/or 0x0a byte. This problem makes it really obvious that text logs are binary files.
See the format definition at https://www.freedesktop.org/wiki/Software/systemd/journal-fi...
And here, this isn't perfect but if you had to hack out the text with no journalctl available you could try this:
grep -a -z 'SYSLOG_TIMESTAMP=\|MESSAGE=' /var/log/journal/69d27b356a94476da859461d3a3bc6fd/system@4fd7dfdde574402786d1a1ab2575f8fb-0000000001fc01f1-0005c59a802abcff.journal | sed -e 's/SYSLOG_TIMESTAMP=\|MESSAGE=/\n&/g'
Thanks for pointing that out. I guess thats why they came up with their own format instead of just using sqlite, or something else that is already a standard.
Everyone forgets or tries to ignore that text files ARE A BINARY FORMAT
That's a bit pedantic, even for HN standards :-)
But yes, I know all about fragile log parsers and race conditions of multiple processes writing to the same file. I was just thinking about a scenario where you end up having to read raw logs when things go haywire.
This is exactly what I don't understand. This is a world where no other computer exists? I have bigger things to worry about (even if you scope the problem down to to "no other computer with journalctl installed exists on my network").
> Or if you're writing to network storage and would like to analyze the logs from your haiku box.
I don't have any Haiku boxes, but I do have Windows boxes. At my day job, where I'm a Linux sysadmin for a large finance company, my workstation is Windows, and I don't have admin on it. So even with conventional UNIX logs, a much more common case is - as I mentioned - that I'd want to read them from a computer that doesn't have gzip installed.
But we're fine with gzip logs, because the way I'd actually do this is to get a Linux computer running.
(Also, keep in mind that the filesystem itself is a binary format. If you're really worried about reading logs from Haiku, you wouldn't put /var/log on NFS because that sounds like a terrible idea, you'd log to a FAT filesystem. But nobody actually does that. Everyone's logs are on ext4 or XFS or btrfs or whatever, and nobody says those formats are a bad idea.)
What about a windows user that has a smart device of some sort that is acting up. They're able to dump the files, but now they have to read them. It's a more realistic scenario than a haiku box, but its the same idea.
And for the record, I am against gzip logs too. I think if you need to save that many logs you should be exporting them to another system instead of archiving them locally.
Playing devil's advocate: system init by itself is easy, just have a single script starting each daemon in sequence, like it was done in the distant past (IIRC, "init" started both the getty for each terminal, and ran a single startup script). It's the "service restart, logging, dependency management" part that's complicated. And unfortunately, since nowadays devices are often hot-pluggable, you can't really escape from the "dependency management" part.
It's not. Read the shell scripts that openbsd uses to init. Simple, straightforward, easy to understand.
I read your comment more as a tribute to the excellent work of the OpenBSD team than a denial of the thesis of the complexity of the init process.
That is, people who underestimate the difficulty of a project are more likely to attempt it, and people who don't understand the area well are more likely to underestimate its difficulty.
« How likely is it that the prolific systemd team and tech decision makers in Linux distributions don’t have a design instinct and came up with this ball of mud full of accidental and unneeded complexity? »
is likely to be fruitful.
If people want to discuss whether systemd is well designed, it would be better to look at the design directly.
The systemd way seems to be "one tool of complexity 40 to 50 things" rather than "50 tools of complexity 1 each doing one thing".
When you only need 10 things, you only need 10 simple tools of complexity 10, rather than one tool of complexity 40.
Though, I see unfulfilled urge to give Linux "serious enterprise grade" twist, and bog everything in "serious enterprise frameworks of doom"
I thought you were the same as the other person. You're continuing the same argument, so it really doesn't matter if you're the same or not: "quickly made systemd a hard requirement" is bullshit. Further, it could easily be worked around.
If you notice e.g. the history of Ubuntu it happens regularly that you hold back a component if there's a problem integrating it. This happens across multiple distributions. It isn't something unique, nor special.
GNOME nor Red Hat did NOT "quickly made systemd a hard requirement". Gentoo was great to ensure that the API that was depended upon was implemented separately. Aside from that, a distribution could also hold back the logind change that caused this change. Skipping over all of these details is great to make this into some big conspiracy story. However, it is rewriting history. It wasn't something unique. Yeah, GNOME release team misjudged one thing. But it actually took a few years before it became an issue. Not this drivel with "OMG they added a hard dependency". It wasn't like that.
AFAIK the GNOME Wayland session still depends on logind, but that's more because there has been no interest in getting it to work on BSD yet.
It applies to both. We need desktops to boot up fast, because you said it yourself, sometimes they just need to. And no one likes waiting around for their machines to boot. Can you imagine the volume of complaints about long boot times that would come in to large-scale distros from annoyed users? That alone makes it a high priority.
And on top of that, we need servers to boot up fast, because nowadays they're virtualized and started/stopped constantly when services are scaled up and down. Can you imagine trying to scale up a fleet of servers and waiting a couple of minutes for each one to boot?
I didn't say that. Because they don't.
> And no one likes waiting around for their machines to boot.
Nobody cares, if it's once for every month or two. Which it is.
> we need servers to boot up fast
But it's not actually booted until the service is up, so it's moot.
Yes, you did. Here's a refresher:
> My workplace even had to install corp software that forces a reboot every N days (with warnings ahead of time) because people just Do. Not. Reboot.
I.e. sometimes computers just need to reboot, and there's nothing you can do about it.
> Nobody cares
If nobody cares, then why do people hate rebooting so much?
With per second billing, fast boot times save money and enable lower fixed capacity, further lowering cost.
As a business/ management matter, we frequently cannot do the technically obvious thing due to other constraints.
>> You can’t create a simple solution for that, since there is an inherent complexity.
They didn't bury the complexity, they removed it. And I agree, that's hard to do. It'd be nice if the systemd folks put in the same effort to remove the complexity from their system.
A standalone (and cross-platform) journalctl file reader seems like a useful thing to have around and not a terribly difficult thing for someone to build.
That's a very emotional take on the whole thing. As I saw it, systemd happened and the Ubuntu developers eventually concluded "well, that's better than upstart, let's use that." Plenty of other distros made the same rational decisions. Meanwhile, the GNOME developers thought "great, people are converging around a modern init system, we can actually integrate with it now," and so they did.
Also, GNOME is not under Red Hat's control. They contribute a lot, but the leadership rarely has a majority of Red Hat employees. While a large number of contributors work there (of course they would - Red Hat is big!), the majority are - again - from elsewhere. I can think of plenty of recent features that people assume are Red Hat driven and I can assure you they definitely are not.
What your take is doing is discounting a very large number of peoples' wisdom, time, and effort, by claiming their contributions are made as helpless victims of some conspiracy or as evil supporters of it. Both of these ideas are harmful.
> plenty of recent features that people assume are Red Hat driven and I can assure you they definitely are not.
My claim is specifically that the people who added the hard dependency on systemd to GNOME were Red Hat employees. I'm not talking at all on who wrote or merged any other code in it.
What utter drivel. I've participated in the discussions around systemd in various distributions. There was a huge amount of discussion, then one by one distributions switched. Some quickly, some took various years. Again, some distributions took various years to switch.
That you can only say things such as "forced down our throats" and "duress" says enough. Not capable to actually hold a discussion, let's be emotional and without any actual facts.
Does this sound any better? Red Hat used their influence over GNOME (and other programs) to add a hard dependency on systemd to it. This forced other distros to either switch to systemd or drop support for GNOME. I suspect that had Red Hat employees not added hard dependencies on systemd to any other software, that no distributions other than Fedora and RHEL and its clones would require it.
Further, it wasn't even a hard dependency. You're really not understanding components and APIs.
> This forced other distros to either switch to systemd or drop support for GNOME.
No, again entirely incorrect. GNOME runs without systemd. A few distributions worked on ensuring GNOME runs without it. It took a while to make that happen, so for a bit some distributions needed to keep some components back. But still: you're talking about systemd while it was an interaction of a few components. Systemd consists of loads of bits.
GNOME _runs_ on distributions without systemd! It took work to make that happen, we coordinated to ensure the problems would be solved.
> I suspect that had Red Hat employees not added hard dependencies on systemd to any other software, that no distributions other than Fedora and RHEL and its clones would require it.
Again, you're so incorrect it's not funny. Arch was really quick to switch to systemd. I help out with Mageia, they really wanted to switch as well, but it took (volunteer) time to make it happen. Opensuse took a while, but still, they would've switched.
The only unique ones were Ubuntu (political crap) and Debian (partly due to political influence by Ubuntu).
Systemd was selected on merit by loads of distributions, not this conspiracy thing you're pretending it to be.
People are conflating systemd itself with all the optional services it comes bundled with. I don't even believe resolved and networkd are enabled by default, at least not by the vendor. Whether a distro enables it depends on the distro.
So now it's not about startup time at all?
> please consider submitting bug reports and patches to help the project,
Who says I don't?
But systemd needs a few full time adults, not just a patch here and there to fix the launch-while-brainstorming culture that gave us the current situation.
>But systemd needs a few full time adults, not just a patch here and there to fix the launch-while-brainstorming culture that gave us the current situation.
Then start working it on it full time, and get some of your friends hired too? Surely you can find someone to pay for that, if it's useful? What else is it that would satisfy you here? I'd be happy if another group was committing full time to systemd (or another similar open source project) just to fix bugs. I fully support you if you decide to do that.
I have told you multiple times why not.
I have, repeatedly. But like in this thread you just reply with the same question asked one more time:
https://news.ycombinator.com/item?id=27653716
> People are still running Linux in memory constrained environments
So why do they have all these memory-hungry services they don't need on standby?
Does that mean that I can DoS these machines simply by connecting to all the open ports, thus starting up the heavy daemons in the constrained environment?
Why is that a good thing?
>> You check if the service is turned on (systemctl or whatever), you check if the process is running (ps, etc), and you check listening ports (netstat, nmap, etc...), they all agree that the service either is or isn't running. And if it's running, it's successfully run its initialization and should be usable.
> This isn't really correct, netstat or nmap won't show process status at all.
This is HN, not reddit, so I'm going to assume you're not just trolling.
netstat -na | grep tcp.*443
Same argument with nmap.
Compare this with the fd-passing model, where you can have every port on your system bound, and it tells you nothing (while troubleshooting) which services are actually up.
Do you not see how "all the ports are bound" then becomes completely useless in troubleshooting and checking status?
Will it tell you if you're actually running SSH on port 443? No, of course not. That's not how troubleshooting works, like at all.
>Does that mean that I can DoS these machines simply by connecting to all the open ports, thus starting up the heavy daemons in the constrained environment?
I'm not sure I'm understanding this question? A lot of machines are not open to the public internet, so this probably doesn't apply there. You can also use some cgroup managing tool (like systemd) to restrict memory usage to the process and configure the OOM killer behavior, so that would also prevent DoS attacks.
> Yes, actually, netstat will show you if you have an HTTPS server running. It will show you if you have an SSH server running.
Actually no, this is wrong, at least for me when I tried the version of netstat that ships with debian. It only shows if something has the port open -- that thing could be an fd holding service (like inted or systemd or something else), or it could be a load balancer, or it could be another service that is incorrectly configured to use the wrong port, etc. So you're right that this complicates the system but this isn't really systemd's fault, and there is nothing that a service manager can really do about this. The only way to know for sure is to use a different tool that prints information about the owning process -- that way you know for sure if it's sshd or something else. Maybe you have a version of netstat that shows this information? If so, then it's not a problem at all, just simply check that column before you continue with your trouble shooting.
>Will it tell you if you're actually running SSH on port 443? No, of course not.
Well now you got me confused, this seems to be directly conflicting with when you said this: "It will show you if you have an SSH server running"
An internal audit is enough to trigger it. "Port scan crashes machine" is not exactly "reliable software".
> You can also use some cgroup managing tool (like systemd) to restrict memory usage to the process and configure the OOM killer behavior, so that would also prevent DoS attacks.
But that means that the default is bad, and unsuitable for resource constrained machines. Which circles back to "neat, but no actual use case".
> Actually no, this is wrong, at least for me when I tried the version of netstat that ships with debian. It only shows if something has the port open -- that thing could be an fd holding service
So you agree that it's a bad idea?
> So you're right that this complicates the system but this isn't really systemd's fault
It is, because it's needless complication. At least inetd was a model to make things simpler. It's the cgi-bin of network services.
But you'll notice that people don't write inetd-based services anymore. In fact my Ubuntu default install doesn't even have inetd installed.
> The only way to know for sure is to use a different tool that prints information about the owning process
netstat has supported this for (maybe) decades on Linux. It's the -p option.
But aside from systemd's poor choices if you see port 22 open, then you can actually be very sure that there is an sshd running, that successfully started (not too broken config).
You could still be wrong. Someone could have started netcat there, or just a honeypot, or whatever, but you can't tell me it's not useful information.
> Well now you got me confused, this seems to be directly conflicting with when you said this: "It will show you if you have an SSH server running"
… unless systemd broke this functionality. I'm making the point why it's a bad idea to break this.
Clients connecting will also not get useful error messages. Port is closed means service not running. Timed out waiting for SSH banner means something else.
Pre systemd it was essentially never anything other than inetd that held ports for others. And for about the last 20 years even it would only do things like echo,chargen,time service that people would run. And having those open by default is from a more naive time, where people thought "sure, why not run tftpd and time service, would could possibly go wrong?".
Nowadays they're off by default, because we're more experienced that any attack surface is still an attack surface, no matter how small.
Probably it helped that OpenBSD kept bragging about how many remote holes in the default install. It's not actually because OpenBSD had better code, it was just that a default OpenBSD only had OpenSSH open to the world.
You are not paying per-second for the VM. The VM itself adds zero value to you. It's the service that's running (or in this case, not) that you're paying for.
Who cares how long it takes before systemd calls listen()? Nobody derives value from that. You're not paying for that. You're paying for the SERVICE to be ready. And if you're not, then why are you even spinning up a VM, if it's not going to run a service?
It's just clever accounting.
And this is the attitude that brings us shitty software, and "I dunno, just reboot to fix the problem?", which is what we have now.
Short of kernel upgrades they really really don't.
But if you've bought in to "oh computers just need to reboot sometimes", then I guess you fall into the category of people who have just given up on reliable software, or you don't know that there is an alternative and no this was not normal.
>>> We need desktops to boot up fast, because you said it yourself, sometimes they just need to
>> I didn't say that. Because they don't.
> Yes you did[…] > people just Do. Not. Reboot.
So is that what I said? I believe you did not read what you quoted.
The main reason people reboot is because of shitty software that requires reboots. So if you want to go self-fulfilling prophecy, then systemd is optimizing for boot times because it's low quality software that requires periodic reboots?
But maybe you count forced reboots once a month (or every two months) for kernel upgrades (but also the above arguments since they also run systemd and therefore need reboots). Fine.
So in order to save ten seconds per month (from a boot time of a minute or so, including "bios" and grub wait times, etc.., so not even a large percentage) this fd-passing silently breaks heaps of services, wasting hours here and there? And that's a good idea?
And all for what? Because you chose to have installed services you don't need, and don't use? And if you do use them, then the time was not saved anyway, but just created a second wait-period where you wait for the service to start up?
And ALL of these services could in any case be fully started while you were typing your username and password.
So what use case exactly is being optimized? The computer was idle for maybe half the time between power-on and loaded desktop environment anyway.
> If nobody cares, then why do people hate rebooting so much?
Because all their state is lost. All their open windows, half-finished emails, notepad, window layout, tmux sessions, the running terminal stuff they don't have in tmux sessions, etc… etc…
This is the key point you are refusing to hear. No, all of the services on a modern Linux machine can't be started while you're typing in your credentials. So they're started lazily, on-demand, one of the classic techniques for performance optimization and a hallmark of good engineering.
How long do you think it takes to start gpsd, or pcsd? Even my laptop has 12 CPU threads, all idle during this time. And including human reaction time (noticing that the login screen has appeared) this is, what, 10 seconds? 120 CPU-seconds is not enough? All desktops run on SSD now too, right?
In fact, how many services do you even think are installed by default?
And Linux, being a multitasking OS, doesn't even have just that window.
But you know, maybe it's a tight race. You could try it. How long does it take to trigger all those?
> a hallmark of good engineering.
In the abstract, as a "neat idea", yes. In actual implementation when actually looking at the requirements and second order effects, absolutely not.
You know you could go even further. You could simply not spin up the VM when the user asks to spin up a VM. Just allocate the IP address. And then when the first IP packet arrives destined for the VM, that's when you spin it up.
That's also a neat idea, and in fact it's the exact SAME idea, but it's absolutely clearly a very bad idea[1] here too.
So do you do this, with your VMs? It's cleary "started lazily, on-demand, one of the classic techniques for performance optimization and a hallmark of good engineering".
[1] Yes, very specific environments could use something like this, but as a default it's completely bananas.
I'm not sure I understand where you're coming from here? I explained how it could be made suitable, it could be done in a way that was crash resistant. I don't know if it's a bad idea or not, it depends on what you're trying to accomplish. The default here is configured by the distro, so you could expect to see a different default on an embedded distro.
>In fact my Ubuntu default install doesn't even have inetd installed.
I believe this is mostly because systemd has replaced its functionality.
>netstat has supported this for (maybe) decades on Linux. It's the -p option.
Good call, I forgot about that, I always use lsof. But that's exactly what I mean, it will show you which pid has the port open, so it will make it obvious if it's systemd or sshd. You won't be sure if there is actually an sshd running unless you check that. This really seems like a non-issue, you have all the tools you need to troubleshoot it.
>systemd broke this functionality. [...] Port is closed means service not running. [...] Pre systemd it was essentially never anything other than inetd that held ports for others.
I don't really want to discuss this anymore if I have to repeat myself, but this is not correct. There are multiple other reasons why you would have another service holding the fd open, such as load balancers, filtering proxies, userspace firewalls, etc, etc. The ability to pass an fd to a child process is an intentional feature of every Unix-like operating system that I've used. Systemd is only using the feature as the OS intended it, which is also supported on OpenBSD.
The fact that it took _work_ to get GNOME to run without systemd is a bad thing in my book.
Also, to claim that systemd was selected on merit without anything backing up the claim of "merit" is disingenuous; plenty of worse solutions end up winning all the time.