we run a ~70 node system, ~95% virtual/5% physical, almost fully converted from Scientific Linux 6 to CentOS 7 (only the NFS boxes left!), which is systemd based.

Concerning systemd as an init system, it's been extremely reliable and simple to use and manage. Writing unit files, even ones with complex requirements, is incredibly easy, because the FDO documentation is really terrific. For instance, we have a host running docker for services that are contingent on an NFS mount being present on the host: specifying a requirement on the mount-point for the docker service to boot-up is as simple as creating a docker.service.d directory and writing a 2-line drop-in unit file with RequiresMountsFor=/mount/path. Being able to add this functionality so easily, without having to assume total ownership of managing the rest of the service configuration really struck me as exemplary of the reasons systemd won the init wars. It really feels like it bends over backwards to help administrators get shit done, and done right.

Systemd also provides a lot more visibility into service behavior than other init systems i've worked with (logs tailed in status output, exit codes, real pid tracking, etc), which is very helpful in day-to-day work as an administrator. I'm also pretty regulary surprised by how fast our servers reboot, though that's a rather minor benefit. Mount units work just like it says on the tin, not much to report. We haven't started using any of the other systemd stuff like timers, but may do so as we encounter use-cases where it's preferable.

On the flip side, I'm not especially keen on journald. It makes it pretty rough to do system log aggregation out of the box. There's some plans to add the ability to forward logs to a remote server, but it's not there yet on CentOS. Rsyslog has nice integration on CentOS that can help make things resemble a more traditional log structure, but I can't speak to it's viability as we ended up using a more lightweight solution that uses journalds option to forward all message via syslog over a local socket (/run/systemd/journal/syslog) as an input for our log aggregator. It's irritating on occasion, but we've never not been able to work around it, and i'm confident that things will get better going forward, as journald gets more features and more log aggregators add support for polling the journal.

most importantly, i am really looking forward to Ubuntu 16.04 coming out with systemd, at which time Ubuntu, Debian, CentOS and more will all use the same init system in an LTS release. as a regular author of open-source chef cookbooks, it's exciting to think of a future where there's enough x-platform consistency that it's no longer necessary to special-case service management. along this same line, systemd unit aliases are also pretty awesome, and i hope package maintainers in both the rpm and deb world start working to add aliases for divergently named services (apache2 vs httpd, mysql vs mysqld, etc.).

ultimately, we're not dogmatic about the philosophy arguments, most of which at this point strike me as sour grapes. i just want a system that works reliably, is well documented, and makes it easy for me to get shit done. systemd fits that bill better than its predecessors, so we're happy using it until something better comes along :)

user:	nathwill
created:	November 7, 2013
karma:	30