Essential security for Linux servers (2013)

Essential security for Linux servers (2013)(sollove.com)

199 points by jcgoette 4 years ago | 146 comments

acatton 4 years ago |

Even for 2013, this is not very good advice. fail2ban makes your iptables dynamic, which is a nightmare to audit.

This is what I do in 2021:

* Set up spiped[1] in front of SSH

* Install and setup nftables[2].

* Lock down every service as much as possible in systemd[3]. (If the service is built-in the distro, just use drop in files[4])

[1] https://www.tarsnap.com/spiped.html

[2] https://wiki.archlinux.org/title/nftables

[3] https://ruderich.org/simon/notes/systemd-service-hardening

[4] https://wiki.archlinux.org/index.php?title=Systemd&oldid=704...

hnlmorg 4 years ago | |

I wouldn't say fail2ban is bad advice per se. The kind of companies running deployments that do auditing on iptables are going to be very different to the types of companies that benefit from dynamic rules created by services like fail2ban.

There is a gulf of difference between hardening a Linux server for an independent web shop vs running Linux at Google. And this article very much feels like it's aimed at sysadmins of the former rather than SRE's of the latter (the fact that they're not even running configuration management like Puppet is a dead give away of that fact)

acatton 4 years ago | | |

> I wouldn't say fail2ban is bad advice per se. The kind of companies running deployments that do auditing on iptables are going to be very different to the types of companies that benefit from dynamic rules created by services like fail2ban. […] There is a gulf of difference between hardening a Linux server for an independent web shop vs running Linux at Google.

I agree that there is a difference between running Linux at a FAANG and running Linux for an independent web-shop.

However, my advice was targeted at the hobbyists like me who likes to run their own webserver. (Independently from my employer) And I think it is appropriate for independent web-shop as well.

Auditing is not just reserved to bigcorps, I personally like to log diffs between "nft list ruleset" and "cat /etc/nftables.conf" on my personal servers. If you run fail2ban this becomes impossible.

Also, IMHO, fail2ban doesn't really solve the problem, a botnet attack could try to bruteforce your SSH. All it does is try to prevent one person from trying too much, it can also lock you out during an emergency. spiped is, IMHO, easier to setup and cheaper to maintain. It also provides a higher degree of protection. (As I explained before, it is a 256bit combination port knocker)

I personally think that fail2ban is a cargo-cult.

bombcar 4 years ago | | |

fail2ban basically just cleans log files for you - and gives someone else control over your iptables. And in the days of botnet attacks, it doesn’t do as much as someone might think.

Moving the ssh port (even without port knocking) does a lot more to cut out log messages.

Or make ssh IPv6 only.

johnisgood 4 years ago | |

> The simplicity of the code — about 6000 lines of C code in total, of which under 2000 are specific to spiped (the rest is library code originating from kivaloo and Tarsnap) — makes it unlikely that spiped has any security vulnerabilities.

spiped might be great, but I found the above on their website. The fact that it has 6k lines of code does not mean that it lacks security vulnerabilities... at all. It does not make it that unlikely either. You still have to audit it. Less LOC just means it will consume less time to do that, but it is of no guarantee that it is more unlikely to have security vulnerabilities.

Plus they could have used better crypto.

matthewaveryusa 4 years ago | | |

Agreed and, fundamentally, why not just ssh -L?

tyingq 4 years ago | |

There are also pam modules that can dynamically block repeated failed ssh login attempts. Pam_shield, for example defaults to blocking by null routing the ip, but you can drop in whatever action you want. There are other similar pam modules as well. I like a pam based approach since it isn't trolling log files, but directly controlling the auth.

iso1631 4 years ago | |

Why use spiped rather than wireguard?

acatton 4 years ago | | |

Wireguard is also a good alternative. But you need to connect to a VPN every time you want to SSH.

Here is my personal reasons why I use spiped:

* spiped is transparent by using ProxyCommand[1]. This allow me to do "ssh host" and thanks to my ssh_config, it just connects.

* spiped can be run in a very hardened way.[2] It just needs to listen() to a socket, connect to another one, and access a key file. Wireguard needs complex network access, it needs to create interfaces and open raw sockets.

* spiped is much simple to manage, just run a daemon. With wireguards there are two possibilites:

** Every host runs wireguard, you might need to connect to multiple hosts at the time, you need to manage internal IP conflicts, etc...

** One central wireguard server, you have a single point of failure, and can't ssh anywhere if this host is down.

Don't get me wrong, I love wireguard, use it all the time as a VPN, but I don't think it's appropriate as a layer of protection in front of my SSH server.

Both Wireguard and Spiped are written by very smart people.

[1] https://man.openbsd.org/ssh_config#ProxyCommand

[2] https://ruderich.org/simon/notes/systemd-service-hardening

discardedrefuse 4 years ago | | |

Not OP, but I was wondering why use spiped instead of simply sshd with passwords disabled. If anyone else was curious, this is what I found on the spiped website:

"You can also use spiped to protect SSH servers from attackers: Since data is authenticated before being forwarded to the target, this can allow you to SSH to a host while protecting you in the event that someone finds an exploitable bug in the SSH daemon -- this serves the same purpose as port knocking or a firewall which restricts source IP addresses which can connect to SSH."

BenjiWiebe 4 years ago | |

I wrote a small tool that's similar to fail2ban. I had it put the IPs into an ipset, and had my firewall rules static. Just had a rule to match the ipset.

marvinblum 4 years ago |

Everytime I read something about Linux server hardening, I get more confused. We're lacking a clear and simple, modern guide on how to do things. I know, every setup is different, but there should at least be consensus for a fresh installation.

Also, do I really HAVE to change something so that it is secure? Isn't a Ubuntu server secure out of the box? With a strong, unique root password of course.

devwastaken 4 years ago |

Do not lock down to your IP address. Home IP's change all the time, and in proper security there should be no other way to access the server other than proper ssh authentication or physical access. There is no good reason to be doing it in this context. If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down.

abdusco 4 years ago |

What has helped me more than fail2ban with reducing login attempts by many orders of magnitude is changing default SSH port from 22 to something in 10000-30000 range.

belter 4 years ago |

After Security...on Performance, Brendan Gregg First 60 Seconds:

https://netflixtechblog.com/linux-performance-analysis-in-60...

INTPenis 4 years ago |

First step is actually passwd -l root for me, I alwasy lock the root password. After ensuring I have a working admin account of course.

Using the root account at all is obsolete imho. Fedora, CentOS and RHEL all allow me to skip setting a root password and just use my admin user.

maxk42 4 years ago | |

That was explicitly mentioned in the article, as was disabling password-based logins entirely.

deadbunny 4 years ago | | |

There is a difference between ssh disabling passwords and local accounts being disabled.

Say ssh disallows password login but I know the root password, if I ssh on to the box as another user I can then su to the root user. If the root user does is locked I can't do this.

hansel_der 4 years ago | |

imho a distinct admin account is better than elevating a useraccount, which also runs a browser.

INTPenis 4 years ago | | |

Yeah sure but it's still better to lock the root account and create a sudo admin account for all root tasks.

errcorrectcode 4 years ago |

Naïve, black-&-white, over-simplified, amateurish advice.

Security measures to deploy depend on the risk level involved, e.g., potential costs of being hacked.

Measures like SELinux, grsec, fwknop, snort, IDS (tripwire or Samhain), HSMs, hw entropy sources, split SSH/TLS, and microservices compartmentalized into VM containers rather than Docker have their place.

heipei 4 years ago |

What I'd like to see is a modern guide for setting up and operating a cluster of hosts that does not rely on any specific provider settings. Say you want to run a cluster of Ubuntu servers, maybe exclusively with a workload scheduler like k8s, maybe with a mixture of nodes, how do you set it up securely and consistently, how to you apply updates, provision users, deploy applications, and how do you centrally log and alert on events (systemd logs, docker logs, auditd). Bonus points if you there are pointers about how compliant that setup is wrt to modern compliance requirements.

I know it's a lot to ask, but maybe there is such a guide available that does not just fall back to talking about provider-specific features (e.g. IAM).

idoubtit 4 years ago | |

ansible. And similar tools, but lack experience outside of ansible. You can see Ansible a a tool to automate the install/configure/update process that you would do manually on a single server. Then you can apply this "playbook" to any server.

You just need an ssh connection to the target servers, with python installed on them. Of course, you have to write rules for setting up a server, provisioning users, monitoring (deploying Prometheus, pushing logs to a central server...). There various plugins for integrating with providers, but the basic features are provider independent.

Ansible is far from perfect (dependency on python, inconsistent syntax, abuse of aliases, missing a strict mode...), but it's rather easy to learn and I've used it successfully (at a small scale).

formerly_proven 4 years ago |

fail2ban is unnecessary if a non-standard port is used. Even a sub-1024 SSH port gets extremely little traffic with spurious login attempts just once per day or every few days and most of these aren't going anywhere (admin:admin). Similarly I don't think for personal servers and the like there is much point in disabling root login, though I disable password auth in SSH as a general rule. A firewall on a server itself should not be necessary in most cases, because unneeded "listen everywhere for everything" services should not be running in the first place. If this is managed by multiple people, the firewall should be external to the server so that the same person who "just wants to run this service for a test real quick" can't "change firewall policy real quick".

rnestler 4 years ago | |

My experience is that even non-standard ssh ports gets hundreds of login attempts on ssh per day.

formerly_proven 4 years ago | | |

I suppose that depends on what you think of as an "login attempt". Is opening a connection a login attempt? I would say it isn't. Is sending some random protocol header a login attempt? Doubtful. Is failing to negotiate a login attempt? Again, I'd say no (most likely a port scanner looking for old/vulnerable servers). Is SSH-1.5-Nmap a login attempt? I don't think so. As we have disabled password authentication, a client can't actually try to do a user/pass login, so what can't happen, isn't.

These things show up, but are completely irrelevant to security.

usr1106 4 years ago | | |

Yes, it has increased quite a bit in recent years. I think still 5 years ago there was basically no traffic.

Justin_K 4 years ago |

Is there an up to date version of this?

3np 4 years ago | |

Two changes I would make:

* use -t ed25519 to generate keys, much more efficient for same security compared to RSA

* don’t use ufw. It easily becomes a big mess and is a pain to manage with ansible. firewalld is a much better high-lever firewall. Preferably with nftables backend.

If you have a bit bigger fleet and manage a CA you could look into using signed SSH certificates instead of public keys. That way you can provision access centrally without adding individual keys to individual servers.

cheshire_cat 4 years ago | | |

Do you switch Debian based systems to firewalld or do you just prefer RedHat based systems?

vbezhenar 4 years ago | |

While I don't agree to this article, from quick glance it does not seem to be obsolete and everything should work on modern Ubuntu.

One thing that's not immediately obvious is that docker does not care about your firewall.

hamburglar 4 years ago | |

It’s all still what I’d call “good advice if you refuse to take some better advice”. The caveat at the beginning acknowledges that this is a pragmatic approach rather than the best approach, and I think in the intervening time I’ve become more convinced that the better approach is the only approach: namely to automate a lot more of these things, which is alluded to at the end.

I’d also ditch the use of any shared credential other than the emergency root password, which should be locked away and not actually known by any people. Your mechanism for syncing ssh pubkeys (which, btw, isn’t specified in the article, which in my experience means it doesn’t really exist :D) on the shared account should instead populate the user keys directory and there should be one logon per user.

shric 4 years ago | |

One thing I'd update for today is use Ed25519 keys instead of RSA

solfox 4 years ago | |

Not by me (author) :)

xcambar 4 years ago | |

AFAICT, the contents is still up to date and solid advice (except IP locks, but YMMV)

clay-dreidels 4 years ago |

Here is my setup, but I’m sure it could use som work.

https://lamda-chops.bearblog.dev/steps-i-take-when-setting-u...

onehair 4 years ago | |

> Enable Automatic Upgrades I can't count with my fingers alone how many times things broke because I `apt upgrade`d without thoroughly reading the change logs. Or even when reading the change logs something still gets past me. Having auto upgrades on a server is not a good idea.

deadbunny 4 years ago | | |

This is why automatic updates are usually only for security updates, not everything.

rspoerri 4 years ago |

maybe read the last discussion on hn: https://news.ycombinator.com/item?id=5316093

jerrya 4 years ago | |

Is there a library of well-designed, well-tested configuration scripts for various servers implementing various tasks?

egberts1 4 years ago | | |

Well-designed?

Well-tested?

There are lots of scripts out there, notably in Git repositories but none attained the dual-status that you seek.

Sosh101 4 years ago |

Do you really need fail2ban if you require keys for ssh login?

rascul 4 years ago | |

It can help cut down on log sizes.

deadbunny 4 years ago | | |

So does logrotate.

ahepp 4 years ago |

Who cares about fail2ban? Unless you're worried about DOS, it seems pretty silly. Similarly with a firewall, assuming you have NAT and a firewall on your router.

simonmales 4 years ago |

TIL about logwatch. Seems handy as I run a couple of VPSs.

pabs3 4 years ago | |

Sounds similar to logcheck. There is also journalcheck for systemd based systems:

https://github.com/trentbuck/journalcheck

makach 4 years ago |

In a professional setting, today, there is much much more that needs to be taken into consideration.

Anyone else found it sus that the OP didn't mention backups?

paradaux 4 years ago | |

I usually wait until I have content to backup prior to setting up backups, and definitely wouldn't be in my mind in the "first 5 minutes."

If you're concerned you will lose 5 minutes of work make use of snapshotting if available.

Of course the first-5-minute title is hyperbolic, but backups are besides the point when you're first setting up and securing a machine.

Sebb767 4 years ago | | |

On the other hand, backups as an afterthought is what leaves you paying ransoms. I prefer to think about backups directly when setting up the machine, since grouping data directories can help a lot with backup strategies later on. Of course, making sure you're the only one on the machine is step 1, but at least I like to set up backups before placing any serious data on the machine, it's a part of the initial setup for me.

garmaine 4 years ago | | |

> Of course the first-5-minute title is hyperbolic

I don't think it is. I've managed a server directly connected to the internet with a US government IP, and it was being port scanned from a Chinese IP within minutes of being turned on. If you are a target, then there is an adversary out there that is patiently waiting for the opportunity to exploit an unpatched vulnerability in new installs, as if your security is otherwise good it might be how they get their foot in the door on your network.

(In our case I really did have a "5 minute plan" to login as soon as the fresh install was booted, setup a firewall, lockdown the ssh server, and install fail2ban ASAP. I'd then check system logs to see if anyone got in before proceeding. Time was of the essence.)

ncmncm 4 years ago |

That is not the whole job on /etc/ssh/sshd_config. You also need

  ChallengeResponseAuthentication no

vaylian 4 years ago | |

why?

yosamino 4 years ago | | |

Because in the most common use-case it allows for the the same functionality as PasswordAuthentication, so if you want to disallow password-based logins, it also has to go. Note that newer openssh version (don't remember how new) renamed this to KbdInteractiveAuthentication. So check your documentation, and double-check everything you read on the internet.

This article is from 2013 after all.