Userland rootkits are lame (2022)(grugq.substack.com) |
Userland rootkits are lame (2022)(grugq.substack.com) |
The context of this post is somewhat important. It is a direct response to a post titled: Symbiote Deep-Dive: Analysis of a New, Nearly-Impossible-to-Detect Linux Threat
Userland rootkits are not “nearly-impossible-to-detect.” They are not novel, they are not impossible to detect, and they are not the pinnacle of hacker techniques.
I felt that it was worth pointing out that the history of userland rootkits goes back a ways and that they were very easy to detect because they rely on proxying all access to the system. If you bypass the hook they use to enter their proxy, they you evade them entirely.
Forensic and incident response guides used to advise using static linked binaries for exactly this reason. There are guides from the 1990s telling people to do this because userland rootkits were an issue (before kernel rootkits everyone used userland rootkits.)
Here is an example from 2013 which points out that you can’t trust any binaries/libraries on the potentially compromised machine and should use statically linked tools. [0]
LD_PRELOAD rootkits are not new and they are not nearly-impossible-to-detect to detect. My post listed a number of ways to detect them, all of which have been known for decades.
[0] https://www.forensicfocus.com/forums/general/trusted-static-...
It’s worth noting that this has a measurable and enormous impact on system performance, because they usually are adding a bunch of strcmp or similar to every incantation of a bunch of different libc calls.
This is why things 'secure boot' and requirement for signed drivers are a thing.
For example, you can patch libc, and by patch I mean there are different ways of patching. a simply binary patch is unreliable because it'd be pinned to a specific version but you can parse the elf symbols dynamically and patch them, even when libc is updated (assuming certain critical symbols are unlikely to ever change) so it becomes reliable. a lot of "static" programs still dynamically load libc.
There is also a more aggressive but easily detectable approach, just patch every elf file on the system so that that the _start/entry always does something to prevent discovery of the malware like messing with the plt so that all function calls are proxied through rootkit code, where the rootkit filters for parameters and return values it doesn't like.
Kernel rookits are great but secure boot is also a thing, if implemented correctly it would be very difficult to beat. you don't even have to go full on rootkit, just a stealthy implant running as root via a hijacked dynamic library (doesn't have to be libc) will do. this isn't windows, it's not like there is a reliable/common way of making sure system libraries are legit (like authenticode/pe signing). The package managers make sure the correct binaries are being installed but they don't have something like sfc on windows (system file checker) that refuses to let the system boot if files are corrupt and/or restore tampered system executables.
If the attacker has modified the environment to present a specific view of system state, bringing your own environment defeats it.
There are tricks which are better than modifying things to hide. For example, there is a race condition between opendir() and readdir() which you can win by using inotify(). Then you can unlink() whatever, wait a while, then link() it back in. During that time it will be deleted and thus invisible to any detection. (I saw a demo of this 12 years ago, so I might be misremembering a bit. I know it used inotify() and unlink())
Only revealing itself to a defender who looks for it sounds like it’ll be working as intended in almost all cases.
Pretending you're running on a remote system https://github.com/viraptor/libremotec
Cheat in games https://github.com/ProfessionallyEvil/LD_PRELOAD-rand-Hijack...
Fixing bugs where the upstream can't do it.
- GNU stdbuf (BSDs use an env-var instead of hijacking libc)
- fakeroot
- gprofng's data collector
- For a long time it was necessary to use LD_PRELOADto hack around a bug in flashplayer because flashplayer used memcpy when it should have used memmove.
Maybe its easy to find rootkits if you know what you are doing and use a statically compiled tcpdump. Do you know how many times I've done a TCP dump on my computer to check for root kits statically compiled or not? 0. I don't spend my time doing that. All it has to do is use less CPU and RAM than my corporate anti virus already does ( which is a lot ) and I'm never going to find it, cause im never gonna look.
[0] https://intezer.com/blog/research/new-linux-threat-symbiote/
You're not going to find rootkits randomly by deciding to check with tcpdump. You're looking for them in the first place, because they tripped some other detector. That's a big prerequisite.
Like "don't get connected to the Internet"? That's a tall order.
A turkey might be hard by size, but sous vide is basically the easy-bake of cooking, takes no effort or creativity during the cook and you can only really impact it before and after.
That's customer service, I think a lot of (all of) our trillion dollar overlords could do a thing or to and learn about providing reliable service.
Being targeted by an adversary is not the same as being hit by random drive-by attacks hitting every IP in the relatively limited IPv4 address space.
If you are a high value target, AV is useless, since the attackers you care about will target you with bespoke mallare, that will not be in an AV database.
I wouldn't say it's a practical approach. Works for a cool demo, sure, but as an adversary I would be hesitant to use this widely.
There are also timing issues that show up, and you can do any number of anti-debugging tricks which would reveal that the environment is being manipulated. Which is an instant red flag.
In general if the attacker is running at the same privilege level, you can probably evade it or at least detect it. I’m somewhat surprised there isn’t a basic forensics tool that automates all of these tests already.
“sus: [-h] [-v] [-V] [-o file] [-O format]
Sus tests for common indicators of compromise using on generic tests for common methods of implementing userland rootkits. It will check for LD_PRELOAD, ptrace(), inotify() and verify the system binaries match the upstream distribution hashsums. It can be used to dump the file system directly (warning, slow!) for comparison against the output of `find`. See EXAMPLES for more.”
Implementation is left as an exercise for the reader.
> Sus tests for common indicators of compromise
There's a lot of stuff that Linux malware tends to do that almost no legitimate program does, this can be incorporated into the tool. Just off the top of my head, some botnet clients delete their executable after launch, in addition to being statically linked, which is an almost 100% guarantee that it's malware.
Check for deleted executables: ls -l /proc/*/task/*/exe 2>/dev/null | grep ' \(deleted\)$'
(not really reliable) check for statically linked running programs: wc -c $(grep -L libc /proc/*/task/*/maps 2>dev/null) 2>/dev/null | grep -v '^0 '
Although a malicious process can just mmap libc for giggles, and also theoretically libc can be named in a way that doesn't contain "libc". A more reliable method is parsing the ELF header in /proc/PID/exe to determine if there's an ELF interpreter defined.
You can also check for processes that trace themselves (TracerPid in status == process id), this is a common anti-debug tactic.
You can also hide the process by mounting a tmpfs on top of it's proc directory, tools like ps ignore empty proc directories due to the possibility that the process has terminated but it's proc directory is still around. This is obviously easily detectable by checking /proc/mounts or just listing empty directories with numeric names in /proc
Another heuristic can be checking /proc/PID/cmdline for two NUL bytes in a row, some malware tries to change it's process name and arguments by modifying the argv array, however they are unable to change the size of cmdline, hence having multiple NUL bytes is a viable detection mechanism. Legitimate programs do this too, but it's rather uncommon.
You can obviously combine these heuristics to make a decision whether the process is malicious, as by themselves they aren't very reliable
However, a few years ago, I discovered that a process inside a container being ptraced could be ptraced by a second process running as root at the host level.[1][2] I don't know if that's been patched away since then, but my assumption at the time was that it meant that the "there can be only one" aspect of ptrace was more of an arbitrary decision, not a hard limit.
[1] https://assets.bishopfox.com/prod-1437/Documents/Guides/2022...
[2] I'm not sure if the "double ptrace" scenario made it into the final document, but it's the same techniques discussed in there, just attach a tracer to the containerized process from inside the container before you attach gdb or asminject.py from outside of the container.
https://github.com/BishopFox/asminject/tree/main/ptrace_scop...
That specific technique only works if root can still load kernel modules, but if I could throw that together with minimal knowledge of the Linux kernel's inner workings, there's probably a sneakier way.