What scripts have you built to stand up a new server? Setting up a new Linux server can take a long time, and it’s very repetitive. I always forget the workflow to set user passwords, harden ssh with my ed_25519 keys, etc. Show us your scripts! |
What scripts have you built to stand up a new server? Setting up a new Linux server can take a long time, and it’s very repetitive. I always forget the workflow to set user passwords, harden ssh with my ed_25519 keys, etc. Show us your scripts! |
https://docs.ansible.com/ansible/latest/getting_started/intr...
Software for Docker containers is built like this: https://12factor.net/
My Raspberry Pis at home are maybe the only exception, in which case I don’t need to harden anything as they’re isolated on my network and run trivial things like a webcam.
I’ve been a Linux sysadmin my entire professional life, and I’ll never go back to VIMing conf files ever again. It’s wonderful.
That said, if you do have to do this, Packer and Ansible still exist.
After the initial config everything goes through k8s. Using Ansible for that as well.
How do you manage k8s?
At first I considered that it would be a good opportunity to learn something like Ansible, but after looking at the getting started docs, I realized I didn't feel like taking the time. So, a bash script!
I wrote a script that assumed I'd just installed a Debian base image. The script installs a couple needed things with apt-get, adds some extra config to /etc/apt, and then does a full upgrade, as well as installing other needed packages.
Next it creates some daemon users & groups, copies in a file system overlay (mostly config files), and then has a list of things to download and install to /opt (stuff that isn't available from a Debian repository). There are a few things I run on it that come in Docker containers, so those are set up (with systemd service files) as well.
Finally it installs duplicity and restores app data that gets backed up nightly to S3 on my original media server box, and sets things up so those backups will happen on the new box as well.
Ultimately I'm not thrilled with it: some of it is fragile (like 'sed' that changes existing config files), and I of course made mistakes as I was writing it, so I had to also take care that the script was idempotent and could handle being run multiple times on the same box without erroring or re-doing what had already been done. I imagine/assume something like Ansible (or Chef or Puppet or whatever) would handle this for me.
But it worked, and was fairly low-effort compared to learning a new provisioning system, so it's fine. Maybe I'll learn Ansible another day.
For reasons that were really good in context, the software aspects of our station builds were based on a set of step-by-step written manual instructions, to configure the Linux distro to "bootstrapping" state for running the huge configuration shell script.
The huge shell script was written to be idempotent. So it could do both the initial heavy configuration and then the same or later versions of it re-run to make any engineering-change adjustments. Including re-run safely remotely, after the machine was deployed on the other side of the globe, in an active production line. If you're comfortable with Bash or Python, a script like this can be easier than shoehorning the problem into the structure of some declarative tool.
(The really good reasons would sound crazy out of context, and you wouldn't think it worked, but we did ultimately have perfect software uptime. And, yes, there were plans (before the Covid chill on our customers and VC) to do a lot of things differently in our next-gen station, including putting most of it in a container.)
Besides bespoke scripts (Bash, Python, Make), I've also used declarative off-the-shelf tools that do a lot of heavy lifting, including doing things with Terraform. (For example, Terraform to allocate a set of AWS EC2, storage, and networking, to simulate a real metal server environment, for developing some infrastructure software.) But, even with those tools, there's always some kind of documentation of it, and often also bespoke scripts.
Regarding remembering all the steps and getting them right... One of the tricks to documentation for things like "pets" servers is to have a canonical internal wiki into which pretty much all useful info that's not otherwise in Git goes. For example, if you go to the AWS Console to add or change something, have the right wiki page open, and update it, including copy&pastes, as you go. (Also, cross-link that with any issue-tracking for the task.) If you keep wiki access low-friction and high-value, it should save you at least as much time as it costs, and be even more valuable to others. (Put loosely, I've seen situations in which a project or a person's job hinges on whether or not one key sentence was captured in a wiki.)
Sometimes, this documentation can later be turned into a script.
Following these lightweight doc conventions, even my personal laptop has a wiki page on how to reconstruct its configuration, and it's kept up-to-date for years.
you know that it won't drift. you can generate images from the same config you apply to a host. you can stamp out the same config across N hosts. you can completely reconfigure a host without reprovisioning it from scratch. you can run slim VMs by mounting the nix store as read only.
it would be great to run an entire company off nixos this way
(Yes, I’m aware that Ansible and Nix exist. I like them. This is really just a hobby project for learning Go).
So far it’s going great. The Neovim-inspired approach of essentially viewing applications and other incidental configurations (like ssh keys) as “plugins to your core system” feels like the right way to solve this problem. And given that Lua can exist as a simple YAML/TOML/JSON-esque collection of fields, but can also drop into proper functions or even quasi-OOP behavior, writing your own “presets” for a given machine config is also trivial. And Go is proving to be a fantastic language for running it all. Really excited to see where this project leads.
I first run a bunch of checks to try and make sure the build will be successful. Then create the instance where cloud-init does a few basics to allow the rest to work. Once the instance is up, the connection in Ansible flips from localhost to the server, mounts the drives, installs everything that needs to be installed and does whatever configuration is needed, and adds records in whatever systems of record need to be updated. The whole process takes about 10 minutes or so (for a single sever), depending on some external dependencies. The time increases as the count goes up. That’s probably something we could solve for, but it hasn’t been a big deal so far.
I did see the "it doesn't bother me" part, but for others looking to speed up mass provisioning don't overlook ansible-pull <https://docs.ansible.com/ansible/10/cli/ansible-pull.html>
It's also handy for keeping systems in a desired state but I use it a lot more for "clone this when you boot" times hundreds of devices. Error recovery is left as an exercise to the reader but mostly a matter of configuring log egress early and having sane health check policies
The other issue we have is our change management process. We need to provide a single update with the status of everything when it’s done. So our process can only go as fast as the slowest server. We could change this, but it would be a political nightmare I don’t want to deal with.
These are all meant for RasPi embedded controls, so they don't handle a lot of security related things that aren't relevant for just a Pi on a private network without ports open.
I set the password with the flasher utility, then have my app server just use Linux authentication so I have fewer things to mess with and more that can be done with standard tools.
Unfortunately MQTT can't do that and the PKI model is hard to set up fully automatically, but almost all routers have guest networks and such, so relying on WPA3 is fine for non-critical stuff.
If I need remote access, I use Zrok.io and avoid having to manage certs myself.
Strongly recommended for those looking for inspiration. Give "ansible-doc --list" a spin! This shows modules, it can filter for existing roles as well.
The code itself isn't that important - declarative YAML, but best practices/patterns can be an art. I don't actually have much published that wouldn't tie my identities together :P
Avoid using Ansible to wrap 'shell' commands with modules of the same name... outside of information gathering, stick with the modules specific to the work. Also avoid using when - may often prefer 'handlers' :)
My JellyFin server, I have a bunch of powershell scripts mostly to reproduce the config of JellyFin components (NGINX, cert for public access, TV listings, automated content scraping, auto-update, library management, etc). I am building a Powershell module which wraps the JellyFin REST API to help with all this. Lastly, the tasks which run on a schedule self-register with Windows Task Scheduler.
I currently have a working design of my host, which has a jail, that hosts a jail within that jail which then hosts another jail which then hosts two more jails of dns and www.
or, otherwise:
Host -> Infrastructure -> NetServices -> Netbox which hosts DNS and WWW
This agent sits on all jails and reports back stats such as networking with auto throttling. Because jails are just a zfs zvol and config away it's has the ability to create too.
All sitting cosy within a TK GUI. On the roadmap is to plug it in to NaviServer and with essence of vanilla JavaScript turn it in to a web frontend.
The first step is to run a bootstrap script, then run a tailor made playbook for each situation.
Those specific playbooks haven't been made public, but I wrote something[0] last year about how to setup an Android development environment using Ansible, and as part of that shared my bootstrap script.
Up to you, but just in you weren't aware: ansible has a `raw:` task which enables one to run anything that the `connection:` transport tolerates, which can go a long way toward getting rid of any extraneous shell scripting
Further toward that goal, I found that PyPy is "untar and go" for every distribution that I've cared about, so even getting python on the machine can similarly be bootstrap friendly
Here is a basic example:
https://www.digitalocean.com/community/tutorials/how-to-use-...
Also, I am obligated to point it out every time I see it: `curl --fail` for life otherwise bash will cheerfully try and execute the 503 html that curl writes out https://github.com/chrisgo/fabric2-example/blob/159ca5ea1844...
Haven't really looked at it much to be honest. Serverless tech has spoilt me.
Here's half of it.
This does also end up having a multiplying effect where if I want to use ansible I can call these scripts directly (which I find often easier than using ansible itself...). Then usually I can have the system set itself up, or at least 90% of the way and while it doesn't save wall time, it saves my time. I can also make specific options for the distro at hand or when I have certain constraints (it is easy to probe for quickly and that can be put in a common file that can be sourced by other scripts). The scripting method is helpful because you're just doing the same design pattern as when programming: containerizing functions, creating flexibility, modularity, and readability (I also highly suggest putting notes in these scripts. Not for others, for you. The more you automate the quicker you'll forget the awesome tricks you found, but you'll be more likely to remember where to revisit). Because like you said "it's very repetitive." When you see that, then you know there's a great opportunity to leverage your programming skills.
I purposefully try to make these scripts require minimal tooling. My main ones are `curl`, `grep`, and `sed`. So I can generally rely on having those on a fresh system. This is really all you need (though you should use these to grab things like `make`).
Pro tip: make a template maker. I have one for a github source. While I can normally just source my common file, there is some benefit for having everything self contained. You won't be able to write the whole script this way but you definitely can get all the boiler plate out of the way which is at least 80% of it, and hey, maybe you could get an LLM to do another 10% for you. Though it I haven't found one that's really that good at bash. (I suspect that this is primarily due to the metric ton of shit bash scripts, since the average one is beyond terrible). Idk why bash scripting is a "lost art" but it is not that hard (for the love of god, use functions).
I also suggest writing some systemd and cron skeletons. These can save a lot of time and really help as you find mistakes or if you want to add extra system hardening. I do find that common implementations do not have the containerization I want (since you're mentioning hardening). You can't always trust the ArchWiki, but it is usually mostly right. An example might be with Fail2Ban, where I don't like to put logs in /var/log/fail2ban/ instead of /var/log/fail2ban.log{,.{1..N}}
I'd share but I don't want to dox myself here but I'm happy to share some bash tips or other quick hacks.
(to be clear, not everything is or should be installed from source. You don't have infinite time and I don't used Gentoo).
> Idk why bash scripting is a "lost art"
I suspect that has to do with the fact that writing or reading it is often an effort in futility, acting as an efficient rabbet hole time trap.
Bash scripting is supposed to act as glue, and it does so extremely poorly. It relies on program writers coding their interfaces correctly to maintain determinism (which many have done very poorly). It makes no guarantees, or warnings about such errors, it makes it quite easy to make these mistakes, and the respective maintainers of the various utilities it may depend on have said these bugs are working as intended, nofix.
Take a look at ldd sometime, I'm sure you'll notice the output has three different types of determinism problems which prevent passing the output to any automation (and having it work thereafter correctly), without first patching the software.
That particular bug was reported in 2016, it was partially fixed in 2018 by PaX, but the maintainers wouldn't pull the fix (from what I read), so PaX forked it. The bugs still exist there today afaik.
For glue to work, you have to be able to make certain guarantees and be able to correctly transform the output in various ways easily. Visibility in all processes is extremely important as well and changes made on one version should continue to work on later versions.
Unfortunately, by externalizing most of the basic functionality to various core utilities, you potentially get different behavior every time you update, and as mentioned it fails resiliency tests; instead becoming brittle. There is also very little visibility without having deep knowledge of the ecosystem.
Ironically, what was described in the Monad Manifesto got this better than anything else I've seen since.
If you can't trust the package manager, how can you trust the install media?
The moment Ubuntu started deceptively poisoning their repo with pre-packaged fixup scripts that would violate security policy by fixing/re-enable snap if it was disabled, and installed the related snap packages (without prompts) instead, was the moment I stopped using Ubuntu for anything professional or production grade.
The package manager is apt, visibility is through apt. If I wanted a snap package, I'd use snap to install it. If you use apt-get upgrade, you shouldn't have to worry about third-party idiocy violating security policy without disclosure or notice and automatically fixing disabled services, and then installing the snap packages.
We don't use snap for a reason, it is unnecessary and broad attack surface that is unneeded.
you can also run nix as a service on other operating systems. https://lix.systems/install/#on-any-other-linuxmacos-system
`lix-installer` needs to run as `root`, attempting to escalate now via `sudo`...
TRACE execute:execute: lix_installer::cli: Execvp'ing `"sudo"` with args `["sudo", "env", "SHELL=/usr/local/bin/bash", "./lix-installer", "-vvvv", "--logger=full", "plan"]`