Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found in the wild(decoded.avast.io) |
Linux Threat Hunting: ‘Syslogk’ a kernel rootkit found in the wild(decoded.avast.io) |
We run a lot of CentOS 5 virtual machines (and some physical ones! ; and some RHEL4! , and a few Fedora core 8 and 4 !!!), with no end in sight... :(
It is a huge concern for the Infra team, a source of many headaches, and we need to go through oops to keep them running, but:
- Clients don't want to move from OLDPRODUCT that requires extremely old php.
- Dev team is not interested in migrating OLDPRODUCT to a modern platform, or even try to put it in a container. Their eyes are turned to the shiny NEWPRODUCT that is seemingly never fully coming to production (only one client has signed for it).
- New clients are still regularly signed on OLDPRODUCT.
- No one in the org wants to pay for for a migration anyway.
- Since some clients have complained about poor apparent security, what was visible was just hidden behind newer haproxy.
Surely this wouldn't take more than 2 weeks: just figure out the install instructions for the old piece of software, rewrite them as a part of a Dockerfile (or similar set of instructions to build an OCI image, there are other options out there, too), setup some basic CI/CD which will execute Docker/Podman/... build command and update any relevant documentation.
I actually recently did that with some legacy software, apart from additional optimizations like clearing yum/dnf cache after installs, it was pretty quick and also easy to do! If you are also in a situation where you need to use EOL software, I don't think there are many other decent options out there, short of trying to reproduce the old runtime on a new OS (as others suggested).
Running the old EOL OS will simply leave you with more vulnerabilities with no fixes in the long term and is generally a really bad idea. How did your security team greenlight that idea? In some orgs out there the devs would be made to stop doing whatever they're doing (outside of critical prod fixes) and would be forced to migrate to something newer before proceeding.
>- New clients are still regularly signed on OLDPRODUCT.
I mean what's the WHY behind that? Why don't even new customers sign on to the new product? Why is the new product not in production? Is that the same reason?
[1] - https://linux-audit.com/increase-kernel-integrity-with-disab...
I know that problem for a thankfully long gone Java internal application, and well... I went with running old stuff in `debian/eol` Docker containers [1]. Turns out you actually can use Docker as a sort of extremely lightweight VM service.
If they're not providing security fixes for RHEL 6's packages, then why does this matter?
(+) or rather too many.
$ du -sh /lib/modules/$(uname -r)
294M /lib/modules/5.10.0-15-amd64
>> vermagic=2.6.32-696.23.1.el6.x86_64 SMP mod_unload modversions
do you know why they say "approximately match"? I thought it had to match exactly so that the kernel accepts to load the module
The greater the difference between the kernel version you compiled for, and the kernel version you are trying to load it on, the greater the chance something you are relying on changed and the module loader cant resolve all the symbols and so it fails.
So saying a kmod has to match the kernel version is good practice but the reality is not quite as strict.
Red Hat has a list of "white listed" symbols that they try to maintain across a major version of RHEL so if your kmod only relies on them and nothing else then it should load on any kernel version within that release. But that's a Red Hat thing, not a Linux kernel thing.
(Tradeoff of runtime DIY symbol resolution / code grovelling being it's more work, and more likely to be crashy).
As a rootkit author you have considerably more flexibility than most module authors who are constrained by "sanity", maintainability, accepted practice and licensing terms.
Which, based on what previous commentator said, is just about 2 more weeks than anyone want to spend on it.
If no one wants to do it, it doesn't matter if it takes 2 days or 2 months, it won't get done.
Sure, lying about some of the details while getting the gist across might be a good idea (e.g. naming a different OS when you actually have Ubuntu), but there definitely needs to be discourse about the circumstances that people are dealing with in reality, instead of everyone pretending that they are on the forefront of the industry with their security and other practices.
The more you think about it, the murkier everything gets - talking about how things were in a company 5 years ago that's now defunct might be the best possible circumstance, but in a currently active org that might also be a way of getting yourself sacked, depending on how juicy of a target it is and how much attention it attracts.
That said, the company has made the choice to use EOL software, their clients have made the choice to use EOL software and it's bad practices all around - honestly, should any of the software be exposed publicly, I'm pretty sure that this person's comments won't be the first to call attention to the setup, since nation X's hackers/crackers might have already ran automated attacks against it anyways.
In summary, bad practices probably deserve to get called out, just so we know what the situation in the industry is like in reality, but hopefully not at too great of a personal expense.
About 1 million CentOS 6 boxes of some description hitting the update servers as of July 2021. Compared with roughly 2 million on v7 and half a million on v8.
Would be interesting to know how those percentages have changed since then.
The end result is usually to force everyone to move to NEWPRODUCT and deprecate OLDPRODUCT violently (Salesforce Lightning vs Classic, etc, etc). Hopefully enough fixes for NEWPRODUCT get done before all the customers leave.
But what can be said, is that even though IBM contributes a lot to open source, not many would claim IBM is a "open-source company" I think, at least when compared to Red Hat.
If IBM put US$1B into 'Linux' 22 years ago - but this was a small part of their operating budget at the time - do we look at absolute or comparative value?
If IBM buys, three years ago, RedHat for US$34B, does that mean IBM is now the biggest 'open source' company? (If so, the next question is obvious.)
Just build the kernel and set the right options, this is for a Dell XPS13: https://github.com/jcalvinowens/misc/blob/main/kbuild/config...
It takes a few hours to whittle it down for a particular piece of hardware, but I've never broken anything on Debian by running kernels built with CONFIG_MODULE=n.
* Edited for clarity
(you must reboot to re-enable module loading)
Useful on servers where specifying all modules to load is practical (netfilter modules are usually the only new modules unless hardware changes). But, on a workstation, doing so will be very frustrating unless you never plug in any new usb devices etc.
If you know what the devices you are likely to plug in, you could just modprobe them all before disabling it.
Edit: I should mention, this will either result in a massive kernel that consumes a lot of memory, or in very little driver support and your machine will not tend to just work when you plug new devices in. Linux has a lot of drivers; there's a reason why it uses modules.
Ubuntu might not be able to distribute said "no module" kernel, but it might run.
https://github.com/c-blake/kslog has maybe a little more color on this topic, though I'm sure there are whole volumes written about it elsewhere. :)
EDIT: But maybe your "game over" point is just that it is kind of a pipe dream to hope to block all concealment tactics? That may be fair, but I think a lot of security folks cling to that dream. :)
~ # zgrep -F CONFIG_MODULES /proc/config.gz
CONFIG_MODULES_USE_ELF_RELA=y
# CONFIG_MODULES is not set
CONFIG_MODULES_TREE_LOOKUP=y$ uname -sr
OpenBSD 7.1
$ du -sh /bsd
22.0M /bsd
Edit: I'm having trouble finding it in the official documentation, but here's a page that describes how to do it on an old version: https://slackwiki.com/ZFS_root_(builtin) and here's what looks like a script to do that on the current tip of master: https://github.com/openzfs/zfs/blob/master/copy-builtin
That's generally called pivoting and has nothing to do with method of persistence of the malicious code.
OP makes a point that certain systems move or have moved away from giving root user the ability to extend/modify kernel code at runtime via kernel modules, my argument is that none of that matters since root user can still extend/modify kernel code at runtime via binary patching.
OpenBSD restricts that ability as well[1]. Neither /dev/mem nor /dev/kmem can be opened (read or write) during normal multi-user operation; you have to enter single-user mode (which requires serial console or physical access to achieve anything useful). Raw disk devices of mounted partitions can't be altered, immutable/append-only files can't be altered, etc.
You can also choose to completely prohibit access to raw disk devices, although that gets annoying when you e.g. need to format an external drive. There is of course still a lot of potential to do harm as root, but it's not as easy to create a persistent threat or resist in-system analysis by an administrator.
> securelevel may no longer be lowered except by init
> The list of securelevel's effects may not be comprehensive.
So yes, it's a nice sandbox that can help prevent accidents, but doesn't sound like something you should rely on for actual defense.
I do think the defense mentality, as evidenced by many comments in this thread, remains a bit too much about "how challenging to make things" rather than the "in theory possibility". Besides binary patching a static kernel as you say, for example, you could have remote hashes of all relevant files a la tripwire, and remote access and programs to check said hashes. If the attacker can detect and adapt to a hash checking pattern then they can "provide the old file" for some purposes/etc. to hide their presence. To do so they have to also write the code to detect/conditionalize. The rationale of this defense mentality seems to hope for a "distribution of attacker laziness" that may at least "help", but sure - it is just a higher, finite bar. And once the work has been done..game over. But I do not mean to belabor the obvious. Anyway, thanks for clarifying your argument.
If you compile old software on a new OS, every single update to that OS has a chance to blow up your compiled old version, so it takes more hand-holding.
Can be worth it at times, but other times it's just easier to firewall and hope for the best.
Most importantly, all of these features and mitigations are enabled by default, and are pretty much invisible to the end user or administrator; and actually easy to use for a developer. Contrast this with e.g. seccomp or SELinux. Google is even hinting "selinux permissive" and "selinux disable" in top 3 suggestions...