I got hacked: My Hetzner server started mining Monero(blog.jakesaunders.dev) |
I got hacked: My Hetzner server started mining Monero(blog.jakesaunders.dev) |
Also could prevent something to exfiltrate sensitive data.
I know we aren't supposed to rely on containers as a security boundary, but it sure is great hearing stories like this where the hack doesn't escape the container. The more obstacles the better I guess.
If the human involved can’t escalate, the hack can’t.
Even if you are an owasp member who reads daily vulnerability reports, it's so easy to think you are unaffected.
At least that's what I think happened because I never found out exactly how it was compromised.
The miner was running as root and it's file was even hidden when I was running ls ! So I didn't understand what was happening, it was only after restarting my VPS from with a rescue image, and after mounting the root filesystem, that I found out the file I was seeing in the processes list did indeed exist.
But podman has also great integration with systemd. With that you could use a socket activated systemd unit, and stick the socket inside the container, instead of giving the container any network at all. And even if you want networking in the container, the podman folks developed slirp4netns, which is user space networking, and now something even better: passt/pasta.
Also rootless docker does not bypass ufw like rootful docker does.
My intuition is that since the SSH server reports what auth methods are available, once a bot sees that password auth is disabled, they will disconnect and not try again.
But I also know that bots can be dumb.
And for basic web sites, it's much better if it requires no back-end.
Every service exposed increases risk and requires additional vigilance to maintain. Which means more effort.
It highlighted the domain: 'jakesaunders.dev' in the address bar in red text.
You have to define a firewall policy and attach it to the VM.
But I am interested in the monero aspect here.
Should I treat this as some datapoint on monero’s security having held up well so far?
No. The reason attackers mine Monero and not some other cryptocurrency isn't the anonymity. Most cryptocurrencies aren't (meaningfully) mineable with CPUs, Monero (apparently) is. There may be others, but I suspect that they either deliver less $ per CPU-second (due to not being valuable), or are sufficiently unknown and/or painful to set up that the attackers just go with the known default.
Trying to mine Bitcoin directly would be pointless, you'd get no money because you're competing with ASIC miners. Some coins were designed to be ASIC resistant, these are mostly mined on GPUs. Monero (and some other coins) were designed to also be GPU resistant (I think). You could see it as a sign that that property has held up (well enough), but nothing else.
If you used the server to mine Bitcoin, you would make approximately zero (0) profit, even if somebody else pays for the server.
But also yes, Monero has technically held up very well.
This became enough of a hassle that I stopped using them.
But yeah it is massively overspecced. Makes me feel cool load testing my go backend at 8000 requests per second though!
But if they do have a vulnerability and manage to escape the sandbox then they will be root on your host.
Running your processes as an unprivileged user inside your containers reduces the possibility of escaping the sandbox, running your containers themselves as un unprivileged user (rootless podman or docker for example) reduces the attack surface when they manage to escape the sandbox.
And isn’t it a design flaw if you can see all processes from inside a container? This could provide useful information for escaping it.
Docker will overwrite your rules when you publish ports.
Do not publish ports with docker. Do not run internal services on the publicly accessible system.
> RandomX utilizes a virtual machine that executes programs in a special instruction set that consists of integer math, floating point math and branches. > These programs can be translated into the CPU's native machine code on the fly (example: program.asm). > At the end, the outputs of the executed programs are consolidated into a 256-bit result using a cryptographic hashing function (Blake2b).
I doubt that you anyone managed to create an ASIC that does this more efficiently and cost effective than a basic CPU. So, no, probably no one is mining Monero using an ASIC.
If they can enslave 100s or even 1000s of machine mining XMR for them, easy money if you set aside the legality of it.
The attack did not and could not compromise or weaken moneros privacy and anonymity features.
Deliberate heat generation.
If it's cold and you're going to be running a heater anyways, then if your heat is resistive, then running a cryptominer is just as efficient and returns a couple dollars back to you. It effectively becomes "free" relative to running the heater.
If you use a heat pump, or you rely on burning something (natural gas, wood, whatever) to generate heat, then the math changes.
I used a rack of GPUs to heat my house for a few years back when gpu mining was decently profitable, and my electricity bill was 3-4x more than with the heat pump - so you have to keep a close eye on the math when you're running at/under profitability.
E.g. on my systemd-nspawn setup with --private-users=pick (enables user namespacing) I created a container and gave it a bind mount. From the container it appears like files in the bind mount created by the container namespace's UID 0 are owned by UID 0 but from outside the container the same file looks owned by UID 100000. Inverted, files owned by the "real" UID 0 on the host look owned by 0 to the host but as owned by 65534 (i.e. "nobody") from the container's perspective. Breaking out of the container shouldn't inherently change the "actual" user of the process from 100000 to 0 any more than breaking out of the container as a non-0 UID in the first place - same as breaking out of any of the other namespaces doesn't make the "UID 0" user in the container turn into "UID 0" on the host.
>Edit: A few people on HN have pointed out that this article sounds a little LLM generated. That’s because it’s largely a transcript of me panicking and talking to Claude. Sorry if it reads poorly, the incident really happened though!
For what it's worth, this is not an excuse, and I still don't appreciate being fed undisclosed slop. I'm not even reading it.
b) if you want to limit your hosting environment to only the language/program you expect to run you should provision with unikernels which enforce it
Except it seems to have done so in this case?
Next year is the 5th year of my current personal project. Ten to go.
Is there ever a reason someone should run a docker container as root ?
An inbound firewall can only help protect services that aren't meant to be reachable on the public internet. This service was exposed to the internet intentionally so a firewall wouldn't have helped avoid the breach.
The lesson to me is that keeping up with security updates helps prevent publicly exposed services from getting hacked.
js scripts running on frameworks running inside containers
PS so I see the host ended up staying uncompromised
Unless ran as root this could return file not found because of missing permissions, and not just because the file doesn't actually exist, right?
> “I don’t use X” doesn’t mean your dependencies don’t use X
That is beyond obvious, and I don't understand how anyone would feel safe from reading about a CVE on a widely used technology when they run dozens of containers on their server. I have docker containers and as soon as I read the article I went and checked because I have no idea what technology most are built with.
> No more Umami. I’m salty. The CVE was disclosed, they patched it, but I’m not running Next.js-based analytics anymore.
Nonsensical reaction.
Nothing is immune. What analytics are you going to run? If you roll your own you'll probably leave a hole somewhere.
But kudos for the word play!
Backend access trivial with Tailscale, etc.
But that alone would not solve the problem being a RCE from HTTP, that is why edge proxy provider like Cloudflare[0] and Fastfy[1] proactivily added protections in his WAF products.
Even cloudflare had an outage trying to protect his customers[3].
- [0] https://blog.cloudflare.com/waf-rules-react-vulnerability/ - [1] https://www.fastly.com/blog/fastlys-proactive-protection-cri... - [2] https://blog.cloudflare.com/5-december-2025-outage/
Backend access trivial with Tailscale, etc.
Public IP never needs to be used. You can just leave it an internal IP if you really want.
DNS is no issue. External DNS can be handled by Cloudflare and their waf. Their DNS service can can obsfucate your public IP, or ideally not need to use it at all with a Cloudflare tunnel installed directly on the server. This is free.
Backend access trivial with Tailscale, etc.
Public IP doesn't always need to be used. You can just leave it an internal IP if you really want.
Free way - sign up for a cloudflare account. Use the DNS on cloudflare, they wil put their public ip in front of your www.
Level 2 is install the cloudflare tunnel software on your server and you never need to use the public IP.
Backend access securely? Install Tailscale or headscale.
This should cover most web hosting scenarios. If there's additional ports or services, tools like nginx proxy manager (web based) or others can help. Some people put them on a dedicated VPS as a jump machine.
This way using the Public IP can almost be optional and locked down if needed. This is all before running a firewall on it.
I use them for self-hosting.
Keeping the IP secret seems like a misnomer.
Its often possible to lock down the public IP entirely to not accept connections except what's initiated from the inside (like the cloudflare tunnel or otherwise reaching out).
Something like a Cloudflare+tunnel on one side, tailscale or something to get into it on the other.
Folks other than me have written decent tutorials that have been helpful.
They also expose kernel interfaces that, if exploited, can lead to the same.
In the end, namespaces are just for partitioning resources, using them for sandboxes can work, but they aren't really sandboxes.
The firewall could run on a piece of dedicated equipment, where it might not be a server, or it could run in a container, on a dedicated computer, which might be the server.
Again, I'm only speaking about what I have experience with in addition to my past experience and have surprisingly found to run well despite thinking I'd never self-host again.
Cloudflare can certainly do more (e.g. protect against DoS and hide your personal IP if your server is at home).
If you plug in a machine at home, it is behind the router, and behind the router's firewall.
If you want more of a firewall locally, something as simple as an EdgeRouter X can get you started easily with this excellent guide: https://github.com/mjp66/Ubiquiti
The nice thing about using cloudflare tunnel, is theres zero ports to expose, ever. The cloudflare tunnel app running on your local machine is what connects out to the internet and takes care of creating a secure connection between cloudflare and your machine.
If you want to forward more than one port to the machine, you could use something like cloudflare to forward to a machine on your home server, and then have the nginx proxy manager or something send the traffic around internally.
It's totally fine to start with cloudflare, and if you aren't already, something like Proxmox (youtube tutorials are pretty quick) gets you up and running and playing pretty quick. Feel free to ask any other questions you like.
One thing I don't really get is why it is "more dangerous" to expose a port on my home IP, versus exposing a port on a Cloudflare tunnel. In both cases, a random user from the Internet can reach my server, and if I host a vulnerable application on that exposed port, it can be exploited. Right?
In order to host my server at home, but keep it outside my LAN, I have been considering having two routers: a "perimeter" router (not sure if that's how it's called) that connects to my ISP, and my normal "LAN" router. The LAN router does not expose anything, as usual. I connect my server to the perimeter router, so that it is in the "DMZ" between both routers. And on the perimeter router, I expose the port to my server. My idea being that if my server gets hacked, it doesn't affect my LAN. A bit like if my server was on a remote VPS.
And then I can run something like proxmox to separate my different services on my server.
But doing this, I expose my home IP instead of a Cloudflare IP, so now I'm concerned that maybe it is a risk? :-)
If you are using Cloudflare's DNS they can hide your IP on the dns record but it would still have to be locked down but some folks find ways to tighten that up too.
If you're using a bare metal server it can be broken up.
It's fair that it's a 3rd party's castle. At the same time until you know how to run and secure a server, some services are not a bad idea.
Some people run pangolin or nginx proxy manager on a cheap vps if it suits their use case which will securely connect to the server.
We are lucky that many of these ideas have already been discovered and hardened by people before us.
Even when I had bare metal servers connected to the internet, I would put a firewall like pfsense or something in between.
If I run vulnerable software, it will still be vulnerable through a Cloudflare tunnel, right?
Genuinely interested, I'm always scared to expose things to the internet :-).
- exposes the port to be available for inbound connections from anyone on the public internet. When we use a web browser, it's outbound first which initiates responses.
- with an exposed port, you are that much more at the mercy of your firewalls ability to protect and defend the open port, which becomes more of a consideration.
- some people take additional security steps to only allow certain IPs to connect to the exposed port if it works for their scenario.
Compared with the Cloudflare Tunnel:
- if it's a website, for example, nothing is open to the public at all. The CF Tunnel (or a similar tool) conencts first outbound to Cloudflare to setup a secure link between your home server.
- having this amount of security can make it harder to connect back to your own server for admin - this is where a tool like Tailscale (also free) can be handy, where you can continue to have full secured access to the server, and the public side only has whatever you want to expose to the public internet.
- if there's a port or service in specific you're looking to sort out feel free to ask.
Network design:
- keeping a server at home outside of your LAN is a good idea, it could be a perimeter router. DMZ can mean exposed to the internet without a firewall.
- if you read the guide I posted above, it's sounds like an exact match for what your'e trying to figure out - it achieves it with multiple VLANS to separate traffic rules. The PDF has some nice graphics to break it out - I wish I had somethign like this when starting out. The concepts described in the PDF should be possible on most equipment that exposes the settings, and while I don't endorse a particular product, the Ubiquiti EdgeRouter X for the $50 or so is very capable as a starting point for what you are after to be the main router. In thet case of adding a dedicated router like this, you would have to switch your modem into "bridging" mode to let this be the main router for everything. Wireless access points can then be individually added to it. Alternatively if something like pfSense interests you, their parent company makes Netgate equipment that a lot of people seem to love. Both are well represented and supported on Youtube to learn from as well.
With the amount of automated bots that port scan looking for anything/everything that's open, as well as scanning DNS records for server IPs that could be targeted, one of the nice patterns of cloud hosting is how application and data servers are hosted behind firewalls of some kind, to effectively be internal.
As for what's exposed to the web, let's say the payload of a website, if there was something vulnerable in the javascript, that could be a weakness hosted anywhere.
Cloudflare can also help achieve this without too much fuss for self-hosted projects, be it personal, and production grade, assuming the rest of the trimmings are tehre.
Oh I see, so that I benefit from the "professional" firewall of Cloudflare, as opposed to my own that I may have possibly misconfigured or forgot to update etc?
Or is there more, like Cloudflare will block IPs that know to come from malicious actors and things like this?
I disrecommend UFW.
firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can.
firewall-cmd --persistent --set-default-zone=block
firewall-cmd --persistent --zone=block --add-service=ssh
firewall-cmd --persistent --zone=block --add-service=https
firewall-cmd --persistent --zone=block --add-port=80/tcp
firewall-cmd --reload
Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports.
Newer versions of firewalld gives an easy way to configure this via StrictForwardPorts=yes in /etc/firewalld/firewalld.conf.
In my own use I have 10.0.10.11 on the vm that I host docker stuff. It doesn't even have its own public IP meaning I could actually expose to 0.0.0.0 if I wanted to but things might change in the future so it's a precaution. That IP is only accessible via wireguard and by the other machines that share the same subnet so reverse proxying with caddy on a public IP is super easy.
> Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway.
The only non-deprecated way of having your Compose services restart automatically is with Quadlet files which are systemd unit files with extra options specific to containers. You need to manually translate your docker-compose.yml into one or more Quadlet files. Documentation for those leaves a lot to be desired too, it's just one huge itemized man page.
In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries.
This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers.
On the other hand, removing exec permissions to /tmp, /var/tmp and /dev/shm is also useful.
Sadly that's more of a duck tape or plaster, because any serious malware will launch their scripts with the proper '/bin/bash /path/to/dropped/payload' invocation. A non-exec mount works reasonably well only against actual binaries dropped into the paths, because it's much less common to launch them with the less known '/bin/ld.so /path/to/my/binary' stanza.
I've also at one time suggested that Debian installer should support configuring a read-only mount for /tmp, but got rejected. Too many packaging scripts depend on being able to run their various steps from /tmp (or more correctly, $TMPDIR).
If this weren't the case, plenty of containers could probably have a fully read-only filesystem.
On ufw systems, I know what to do: if a port I need open is blocked, I run 'ufw allow'. It's dead simple, even I can remember it. And if I don't, I run 'ufw --help' and it tells me to run 'ufw allow'.
Firewall-cmd though? Any time I need to punch a hole in the firewall on a Fedora system, I spend way too long reading the extremely over-complicated output of 'firewall-cmd --help' and the monstrous man page, before I eventually give up and run 'sudo systemctl disable --now firewalld'. This has happened multiple times.
If firewalld/firewall-cmd works for you, great. But I think it's an absolutely terrible solution for anyone whose needs are, "default deny all incoming traffic, open these specific ports, on this computer". And it's wild that it's the default firewall on Fedora Workstation.
This sounds like great news. I followed some of the open issues about this on GitHub and it never really got a satisfactory fix. I found some previous threads on this "StrictForwardPorts": https://news.ycombinator.com/item?id=42603136.
It’s not like iptables was any better, but it was more intuitive as it spoke about IPs and ports, not high-level arbitrary constructs such as zones and services defined in some XML file. And since firewalld uses iptables/nftables underneath, I wonder why do I need a worse leaky abstraction on top of what I already know.
I truly hate firewalld.
I’d love a Linux firewall configured with a sane config file and I think BSD really nailed it. It’s easy to configure and still human readable, even for more advanced firewall gateway setups with many interfaces/zones.
A have no doubt that Linux can do all the same stuff feature-wise, but oh god the UX :/
Imagine container A which exposes tightly-coupled services X and Y. Container B should be able to access only X, container C should be able to accesd only Y.
For some reason there just isn't a convenient way to do this with Docker or Podman. Last time I looked into it, it required having to manually juggle the IP addressed assigned to the container and having the service explicitly bind to it - which is just needlessly complicated. Can firewalls solve this?
And ufw supports nftables btw. I think the real lesson is write your own firewalls and make them non-permissive - then just template that shit with CaC.
I mean there are some payload over payload like GRE VPE/VXLAN/VLAN or IPSec that needs to be written in raw nft if using Foomuuni but it works!.
But I love the Shorewall approach and your configuration gracefully encapsulated Shorewall mechanic.
Disclaimer: I maintain vim-syntax-nftables syntax highlighter repo at Github.
I'm not even sure what to say, or think, or even how to feel about the frontend ecosystem at this point. I've been debating on leaving the whole "web app" ecosystem as my main employment ventures and applying to some places requiring C++. C++ seems much easier to understand than what ever the latest frontend fad is. /rant
Compare that to what we had in the late 00's and early 10's we went through prototype -> mootools -> jquery -> backbone -> angularjs -> ember -> react, all in about 6 years. Thats a new recommended framework every year. If you want to complain about fads and churn, hop on over to AI development, they have plenty.
Pick a solid technology (.NET, Java, Go, etc...) for the backend and use whatever you want for your frontend. Voila, less CVEs and less churn!
Is that the case, though? My understanding was, that even if I run a docker container as root and the container is 100% compromised, there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?
"CVE-2025-66478 - Next.js/Puppeteer RCE)"
That said, do you have an image of the box or a container image? I'm curious about it.
I was lucky in that my DB backups were working so all my persistence wax backed up to S3. I think I could stand up another one in an hour.
Unfortunately I didn't keep an image no. I almost didn't have the foresight to investigate before yeeting the whole box into the sun!
From the root container, depending on volume mounts and capabilities granted to the container, they would enumerate the host directories and find the names of common scripts and then overwrite one such script. Or to be even sneakier, they can append their malicious code to an existing script in the host filesystem. Now each time you run your script, their code piggybacks.
OTOH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here. The $HOME/.{aws,docker,claude,ssh} Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.
Deleting and remaking the container will blow away all state associated with it. So there isn't a whole lot to worry about after you do that.
Now, JS sites everywhere are getting hacked.
JS has turned into the very thing it strived to replace.
It's good to see developers haven't changed. ;)
If you have decent network or process level monitoring, you're likely to find it, while you might not realize the vulnerable software itself or some stealthier, more dangerous malware that might exploit it.
And maybe updating container images with a mechanism similar to renovate with "minimumReleaseTime=7days" or something similar!?
Then you need to run things with as least privilege as possible. Sadly, Docker and containers in general are an anti-pattern here because they’re about convenience first, security second. So the OP should have run the contains as read-only with tight resource limits and ideally IP restrictions on access if it’s not a public service.
Another thing you can do is use Tailscale, or something like it, to keep things being a zero trust, encrypted, access model. Not suitable for public services of course.
And a whole host of other things.
Easier said than done, I know.
Podman makes it easier to be more secure by default than Docker. OpenShift does too, but that's probably taking things too far for a simple self hosted app.
Re: the Internet.
Re: Peer-to-peer.
Re: Video streaming.
Re: AI.
So... Crypto is illegal so anyone using is defacto a criminal by definition.
Also, for this particular instance this is the best bug bounty program i've ever seen. Running a monero node that hits your daily budget cap is not that bad... It could be way worse like steal you DB creditials and sell it to the highest party... So, crypto actually made this better.
Luckily for me, the software I had installed[1] was in an LXC container running under Incus, so the intrusion never escaped the application environment, and the container itself was configured with low CPU priority so I didn't even notice it until I tried to visit the page and it didn't load.
I looked around a bit and it seemed like an SSH key had been added under the root user, and there were some kind of remote management agents installed. This container was running Alpine so it was pretty easy to identify what processes didn't belong from a simple ps output of the remaining processes after shutting down the actual web application.
In the end, I just scrapped the container, but I did save it in case I ever feel like digging around (probably not). In the end I did learn some useful things:
- It's a good idea to assume your system will get taken over, so ensure it's isolated and suitably resource constrained (looking at you, pay-as-you-go cloud users).
- Make sure you have snapshots and backups, in my case I do daily ZFS snapshots in Incus which makes rolling back to before the intrusion a breeze.
- While ideally anything compromised should be scrapped, rolling back, locking it down and upgrading might be OK depending on the threat.
Regarding the miner itself:
- from what I could see in its configuration it hadn't actually been correctly configured, so it's possible they do some kind of benchmark and just leave the system silently compromised if it's not "worth it", they still have a way in to use it for other purposes.
- no attempt had been made at file system obfuscation, which is probably the only reason I really discovered it. There were literally folders in /root lying around with the word "monero" in them, this could have been easily hidden.
- if they hadn't installed a miner and just silently compromised the system, leaving whatever running on it alone (or even doing a better job at CPU priority), I probably never would have noticed this.
- Hetzner firewall (because ufw doesn't work well with docker) to only allow public access to 443.
- Self-hosted OpenVPN to access all private ports. I also self-host an additional Wireguard instance as a backup VPN.
- Cloudflare Access to protect `*.coolifydomain.com` by default. This would have helped protect the OP's Umami setup since only the OP can access the Umami dashboard. Bypass rules can be created in Cloudflare Access to allow access to other systems that need access using IP or domain.
- Cloudflare Access rules to only allow access to internal admin path such as /wp-admin/ through my VPN IP (or via email OTP to specified email ids).
- Traefik labels on docker-compose files in Coolify to add basic auth to internal services which can't be behind Cloudflare Access such as self-hosted Prefect. This would have also added a login screen before an attacker would see Umami's app.
- I host frontends only on Vercel or Cloudflare workers and host the backend API on the Coolify server. Both of these are confirmed to never have been affected, due to decoupling of application routing.
- Finally a bash cron script running on server every 5 minutes that monitors the resources and sends me an alert via Pushover when the usages are above the defined thresholds. We need monitoring and alert as well, security measures alone are not enough.
Even with all these steps, there will always be edge cases. That's the caveat of self-hosting but at the same time it's very liberating.
$ sudo ufw default deny incoming
$ sudo ufw default allow outgoing
$ sudo ufw allow ssh
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw enable
Meanwhile companies think the clouds are looking at it.... anyhow. it is a real problem.
AWS explicitly spells this out in their Shared Responsibility Model page [0]
It is not your cloud provider's responsibility to protect you if you run outdated and vulnerable software. It's not their responsibility to prevent crypto-miners from running on your instances. It's not even their responsibility to run a firewall, though the major players at least offer it in some form (ie, AWS Security Groups and ACL).
All of that is on the customer. The provider should guarantee the security of the cloud. The customer is responsible for security in the cloud.
[0] https://aws.amazon.com/compliance/shared-responsibility-mode...
Examples like this, are why I don’t run a VPS.
I could definitely do it (I have, in the past), but I’m a mediocre admin, at best.
I prefer paying folks that know their shit to run my hosting.
Yikes. I would still recommend a server rebuild. That is _not_ a safe configuration in 2025, whatsoever. You are very likely to have a much better engineered persistent infection on that system.
The right thing to do is to roll out a new server (you have a declarative configuration right?), migrate pure data (or better, get it from the latest backup), remove the attacked machine off the internet to do a full audit. Both to learn about what compromises there are for the future and to inform users of the IoT platform if their data has been breached. In some countries, you are even required by law to report breaches. IANAL of course.
It has since been fixed: Lesson learned.
This should not enable the compromise of your session, i.e. either there was some additional vulnerability (and not logging out widened the window during which it could be exploited), or the attack was totally unrelated. Either way, you should find out the real cause.
> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.
1. Files of running programs can be deleted while the program is running. If the program were trying to hide itself, it would have deleted /tmp/.XIN-unix/javae after it started. The nonexistence of the file is not a reliable source of information for confirming that the container was not escaped.
2. ps shows program-controlled command lines. Any program can change what gets displayed here, including the program name and arguments. If the program were trying to hide itself, it would change this to display `login -fp ubuntu` instead. This is not a reliable source of information for diagnosing problems.
It is good to verify the systemd units and crontab, and since this malware is so obvious, it probably isn't doing these two hiding methods, but information-stealing malware might not be detected by these methods alone.
Later, the article says "Write your own Dockerfiles" and gives one piece of useless advice (using USER root does not affect your container's security posture) and two pieces of good advice that don't have anything to do with writing your own Dockerfiles. "Write your own Dockerfiles" is not useful security advice.
I actually think it is. It makes you more intimate with the application and how it runs, and can mitigate one particular supply-chain security vector.
Agreeing that the reasoning is confused but that particular advice is still good I think.
/tmp/.XIN-unix/javae &
rm /tmp/.XIN-unix/javae
I noticed this, because Hetzner forwarded me an email from the German government agency for IT security (BSI) that must have scanned all German based IP addresses for this Next.js vulnerability. It was a table of IP addresses and associated CVEs.
Great service from their side, and a lot of thanks to the German tax payers wo fund my free vulnerability scans ;)
Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?
Unless you're running a static html export - eg: not running the nextjs server, but serving through nginx or similar
There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established mechanisms to sandbox themselves it seems.
The way the author describes docker being the savior appears to be sheer luck.
Good security is layered.
I guess you can have the appserver fully firewalled and have another bastion host acting as an HTTP proxy, both for inbound as well as outbound connections. But it's not trivial to set up especially for the outbound scenario.
I know port scanners are a thing but the act of using non-default ports seems unreasonably effective at preventing most security problems.
I did docker pull a few times base on some webpost (looks reasonable) and detect app/scripts from inside the docker connect to some .ru sites immediately or a few days later....
I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.
I don't think it's wrong, it's just not the same as eg using a yubikey.
My sshd only listens on the VPN interface
App servers run docker, with images that run a single executable (no os, no shell), strict cpu and memory limits. Most of my apps only require very limited temporary storage so usually no need to mount anything. So good luck executing anything in there.
I used, way back in the day, to run Wordpress sites. Would get hacked monthly every possible way. Learned so much, including the fact that often your app is your threat. With Wordpress, every plugin is a vector. Also the ability to easily hop into an instance and rewrite running code (looking at you scripting languages incl JS) is terrible. This motivated my move to Go. The code I compiled is what will run. Period.
The core of the problem here is that process isolation doesn't save you from whole classes of attack vectors or misconfigurations that open you up to nasty surprises. Docker is great, just don't think of it as a sandbox to run untrusted code.
Of course if you have a kernel exploit you'd be able to break out (this is what gvisor mitigates to some extent), nothing seems to really protect against rowhammer/memory timing style attacks (but they don't seem to be commonly used). Beyond this, the main misconfigurations seem to be too wide volume bindings (e.g. something that allows access to the docker control socket from inside the container, or an obviously stupid mount like mounting your root inside the container).
Am I missing something?
Docker is pretty much the same but supposedly more flimsy.
Both have non-obvious configuration weaknesses that can lead to escapes.
Second, even if your Docker container is configured properly, the attacker gets to call themselves root and talk to the kernel. It's a security boundary, sure, but it's not as battle-tested as the isolation of not being root, or the isolation between VMs.
Thirdly, in the stock configuration processes inside a docker container can use loads of RAM (causing random things to get swapped to disk or OOM killed), can consume lots of CPU, and can fill your disk up. If you consider denial-of-service an attack, there you are.
Fourthly, there are a bunch of settings that disable the security boundary, and a lot of guides online will tell you to use them. Doing something in Docker that needs to access hot-plugged webcams? Hmm, it's not working unless I set --privileged - oops, there goes the security boundary. Trying to attach a debugger while developing and you set CAP_SYS_PTRACE? Bypasses the security boundary. Things like that.
Unfortunately, user namespaces are still not the default configuration with Docker (even though the core issues that made using them painful have long since been resolved).
non necessary vulnerability per. se. Bridged adapter for example lets you do a lot - few years ago there were a story of something like how a guy got a root in container and because the container used bridged adapter he was able to intercept traffic of an account info updates on GCP
I disagree with other commenters here that Docker is not a security boundary. It's a fine one, as long as you don't disable the boundary, which is as easy as running a container with `--privileged`. I wrote about secure alternatives for devcontainers here: https://cgamesplay.com/recipes/devcontainers/#docker-in-devc...
Are you holding millions of dollars in crypto/sensitive data? Better assume the machine and data is compromised and plan accordingly.
Is this your toy server for some low-value things where nothing bad can happen besides a bit of embarrassment even if you do get hit by a container escape zero-day? You're probably fine.
This attack is just a large-scale automated attack designed to mine cryptocurrency; it's unlikely any human ever actually logged into your server. So cleaning up the container is most likely fine.
Also, if you've been compromised, you may have a rootkit that hides itself from the filesystem, so you can't be sure of a file's existence through a simple `ls` or `stat`.
Honestly, citation needed. Very rare unless you're literally giving the container access to write to /usr/bin or other binaries the host is running, to reconfigure your entire /etc, access to sockets like docker's, or some other insane level of over reach I doubt even the least educated docker user would do.
While of course they should be scoped properly, people act like some elusive 0-day container escape will get used on their minecraft server or personal blog that has otherwise sane mounts, non-admin capabilities, etc. You arent that special.
Search engines try to fight slop results with collateral damage mostly in small or even personal websites. Restaurants are happy to be on one platform only: Google Maps. Who needs an expensive website if you're on there and someone posts your menu as one of the pictures? (Ideally an old version so the prices seem cheaper and you can't be pinned down for false advertising.) Open source communities use Github, sometimes Gitlab or Codeberg, instead of setting up a Forgejo (I host a ton of things myself but notice that the community effect is real and also moved away from self hosting a forge). The cherry in top is when projects use Discord chats as documentation and bug reporting "form". Privacy people use Signal en masse, where Matrix is still as niche as it was when I first heard of it. The binaries referred to as open source just because they're downloadable can be found on huggingface, even the big players use that exclusively afaik. Some smaller projects may be hosted on Github but I have yet to see a self-hosted one. Static websites go on (e.g. Github) Pages and back-ends are put on Firebase. Instead of a NAS, individuals as well as small businesses use a storage service like Onedrive or Icloud. Some more advanced users may put their files on Backblaze B2. Those who newly dip their toes in self-hosting increasingly use a relay server to reach their own network, not because they need it but to avoid dealing with port forwarding or setting up a way to privately reach internal services. Security cameras are another good example of this: you used to install it, set a password, and forward the port so you can watch it outside the home. Nowadays people expect that "it just works" on their phone when they plug it in, no matter where they are. That this relies on Google/Amazon and that they can watch all the feeds is acceptable for the convenience. And that's all not even mentioning the death of the web: people who don't use websites anymore the way they were meant (as hyperlinked pages) but work with an LLM as their one-stop shop
Not that the increased convenience, usability, and thus universal accessibility of e.g. storage and private chats is necessarily bad, but the trend doesn't seem to me as you seem to think it is
I can't think of any example of something that became increasingly often self-hosted instead of less across the last 1, 5, or 10 years
If you see a glimmer of hope for the distributed internet, do share because I feel increasingly as the last person among my friends who hosts their own stuff
I've been on the receiving end of attacks that were reported to be the size of more than 10tbps I couldn't imagine how I would deal with that if I didn't have a 3rd party providing such protection - it would require millions $$ a year just in transit contracts.
There is an increasing amount of software that attempts to reverse this, but as someone from https://thingino.com/ said: opensource is riddled with developers that died to starvation (nobody donates to opensource projects).
I'm glad you're up to writing more of your own posts, though! I'm right there with you that writing is difficult, and I've definitely got some posts on similar topics up on my site that are overly long and meandering and not quite good, but that's fine because eventually once I write enough they'll hopefully get better.
Here's hoping I'll read more from you soon!
> IT NEVER ESCAPED.
You haven't confirmed this (at least from the contents of the article). You did some reasonable spot checks and confirmed/corrected your understanding of the setup. I'd agree that it looks likely that it did not escape or gain persistence on your host but in no way have you actually verified this. If it were me I'd still wipe the host and set up everything from scratch again[0].
Also your part about the container user not being root is still misinformed and/or misleading. The user inside the container, the container runtime user, and whether container is privileged are three different things that are being talked about as one.
Also, see my comment on firewall: https://news.ycombinator.com/item?id=46306974
[0]: Not necessarily drop-everything-you-do urgently but next time you get some downtime to do it calmly. Recovering like this is a good excercise anyway to make sure you can if you get a more critical situation in the future where you really need to. It will also be less time and work vs actually confirming that the host is uncontaminated.
- Configuration management (ansible, salt, chef, puppet)
- Preconfigured images (NixOS, packer, Guix, atomic stuffs)
For a one-off: pssh
You can also restrict outbound connections to cryptomining pools and malicious IPs. For example by using IOCs from VirusTotal or urlhaus.bazaar.ch
With ufw gui you need a single checkbox - block incoming connections.
Perhaps if you're doing more complicated things like bridging interfaces or rerouting traffic it would be more difficult to use than the alternatives, but for a simple whitelist it's extremely easy to configure and modify.
Internet: The Cuckoo's Egg (nation state, more so than criminals maybe, but it's a blurry distinction)
AI: Elon Musk youtube (often cryptocurrency scam) ads
Video streaming was more obviously exclusively a porn thing
Docker also has more traps and not quite as obvious as this. For example, it can change the private IP block its using without telling you. I got hit by this once due to a clash with a private block I was using for some other purpose. There's a way to fix it in the config but it won't affect already created containers.
By the way. While we're here. A public service announcement. You probably do NOT need the userland-proxy and can disable it.
/etc/docker/daemon.json
{ "userland-proxy": false }
Some quick searching yields generic advice about keeping everything updated or running in rootless mode.
{
"log-driver": "json-file",
"log-opts": {
"labels": "production_status",
"tag": "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}",
"env": "os,customer",
"max-size": "10m"
},
"bip": "172.17.1.1/24",
"default-address-pools": [
{"base": "172.17.0.0/16", "size": 24}
]
}Its security through obscurity, which puts you out of view of the vast majority of the chaos of the internet. It by no means protects you from all threats.
Same as for docker, yes?
Networking is just better in podman.
I have been using for many decades both Linux and FreeBSD, on many kinds of computers.
When comparing Linux with FreeBSD, I probably do not find anything more annoying on Linux than its networking configuration tools.
While I am using Linux on my laptops and desktops and on some servers with computational purposes, on the servers that host networking services I much prefer FreeBSD, for the ease of administration.
So you can create multiple addresses with multiple separate "domains" mapped statically in /etc/hosts, and allow multiple apps to listen on "the same" port without conflicts.
..actually this is very weird. Are you saying you can bind to 127.0.0.2:80 without adding a virtual IP to the NIC? So the concept of "localhost" is really an entire class A network? That sounds like a network stack bug to me heh.
edit: yeah my route table on osx confirms it. very strange (at least to me)
You can do:
python3 -m http.server -b 127.0.0.1 8080
python3 -m http.server -b 127.0.0.2 8080
python3 -m http.server -b 127.0.0.3 8080
and all will be available.
Private network ranges don't really have the same purpose, they can be routed, you have to always consider conflicts and so on. But here with 127/8 you are in your own world and you don't worry about anything. You can also do tests where you need to expose more than 65k ports :)
You have to also remember these are things established likely before even DNS was a thing, IP space was considered so big that anyone could have a huge chunk of it, and it was mostly managed manually.
> If your app’s React code does not use a server, your app is not affected by this vulnerability. If your app does not use a framework, bundler, or bundler plugin that supports React Server Components, your app is not affected by this vulnerability.
https://react.dev/blog/2025/12/03/critical-security-vulnerab...
So if you have a backend that supports RSC, even if you don't use it, you can still be vulnerable.
GP said they only shipped front ends but that can mean a lot.
Edit:link
https://nvd.nist.gov/vuln/detail/CVE-2025-29927
That plus the most recent react one, and you have a culture that does not care for their customers but rather chasing fads to help greedy careers.
Also the Docker Compose tool is a well-know exception to the compatibility story. (There is some unofficial podman compose tool, but that is not feature complete and quadlets are better anyway.)
I agree with approaching podman as its own thing though. Yes, you can build a Dockerfile, but buildah lets you build an efficient OCI image from scratch without needing root. For those interested, this document¹ explains how buildah compares to podman and docker.
1. https://github.com/containers/buildah/tree/main/docs/contain...
The docs you need for quadlets are basically here: https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
The one gotcha I can think of not mentioned there is that if you run it as a non-root user and want it to run without logging in as that user, you need to: `sudo loginctl enable-linger $USER`.
If you don't vibe with quadlets, it's equally fine to do a normal systemd .service file with `ExecStart=podman run ...`, which quadlets are just convenience sugar for. I'd start there and then return to quadlets if/when you find that becomes too messy. Don't add new abstraction layers just because you can if they don't help.
If you have a more complex service consisting of multiple containers you want to schedule as a single unit, it's also totally fine to combine systemd and compose by having `ExecStart=podman compose up ...`.
Do you want it to run silently in the background with control over autorestarts and log to system journal? Quadlets/systemd.
Do you want to have multiple containers scheduled together (or just prefer it)? Compose.
Do you want to manually invoke it and have the output in a terminal by default? CLI run or compose.
Of course, when you think about it, nobody expects a command to just survive logging out, but coming from docker, you still have that expectation. And I wonder, am I supposed to be running this on a tmux like the old days? No, you need to do a bunch of systemd/linger/stuff. So being that we are already in systemd land, you keep searching and end up in quadlets, which are a new-ish thing with (last I checked) bad docs, replacing whatever was used before (which has good docs). Docs, being said, that give k8s ptsd. Quadlet, podlet and pods.
It seems that when podman deviates from docker, it does in the least ergonomic way possible. Or maybe I have been lobotomized by years and years of using docker, or maybe my patience threshold is very low nowadays. But this has been my experience. I felt very stupid when I deployed something and it stopped after 5 minutes. I was ready to use podman, because it worked locally. And then it failed in production. Thanks no.
This is not a side effect of running rootless, it's a side effect of running systemd (or rather, systemd-logind).
loginctl enable-lingerI'm going to sit down and rewrite the article and take a further look at the container tomorrow.
(And good to hear you're leaving the LLMs out of the writing next time <3)
At any rate, this happening to you sucks! Hugs from a fellow HN user, I know that things like this can suck up a lot of time and energy. It’s courageous to write about such an incident incident, I think it’s useful to a lot of other people too, kudos!
I tried handwriting https://blog.jakesaunders.dev/schemaless-search-in-postgres/ bit I thought it came off as rambling.
Maybe I'll have a go at redrafting this tomorrow in non LLM-ese.
There is nothing wrong with this article. Please continue to write as you; it's what people came for.
LLMs have their place. I find it useful to prompt an LLM to fix typos and outright errors and also prompt them to NOT alter the character or tone of the text; they are extraordinarily good at that.
Unfortunately, there is no way to specify those `emptyDir` volumes as `noexec` [1].
I think the docker equivalent is `--tmpfs` for the `emptyDir` volumes.
Having such a nice layered buildsystem with mountpoints, I'm amazed Docker made readonly an afterthought.
While LX-branded zones were a really cool tech demo, maintaining compatibility with Linux long-term would be incredibly painful and you're bound to find all sorts of horrific bugs in production. I believe that Oxide uses KVM to run their Linux guests.
Linux has always supported nested namespaces and you can run Docker containers inside LXC (or Incus) fairly easily. Note that while it does add some additional protection (in particular, it transparently adds user namespaces which is a critical security feature most people still do not enable in Docker) it is still the same technology as containers and so kernel bugs still pose a similar risk.
https://www.statista.com/statistics/420400/spam-email-traffi...
That's the point, its private by design and unless they tell you, nobody will ever know how much they use and for what. The true hacker spirit.
If you bother to look past news headlines you will find a vibrant community of people paying for legal goods that value privacy before FUD and ignorance.
This kind of fearmongering is already leading us towards a cashless society because "only criminals use it". This is hackernews and not facebook or congress so it should be obvious to everybody here what the end result of criminalizing/demonizing non KYC payments will be (hint: look at china).
That page does not address rootless Docker, which can be installed (not just run) without root, so it would not have the ability to clobber firewall rules.
In this model, hosts don’t need any direct internet connectivity or access to public DNS. All outbound traffic is forced through the proxy, giving you full control over where each host is allowed to connect.
It’s not painless: you must maintain a whitelist of allowed URLs and HTTP methods, distribute a trusted CA certificate, and ensure all software is configured to use the proxy.
Just use a firewall.
The firewall is there as a safeguard in case a service is temporarily misconfigured, it should certainly not be the only thing standing between your services and the internet.
Attacker now needs a Docker exploit and then a VM exploit before getting to the hypervisor (and, no, pwning the VM ain't the same as pwning the hypervisor).
Not only does it allow me to partition the host for workloads but I also get security boundaries as well. While it may be a slight performance hit the segmentation also makes more logical sense in the way I view the workloads. Finally, it's trivial to template and script, so it's very low maintenance and allows for me to kill an LXC and just reprovision it if I need to make any significant changes. And I never need to migrate any data in this model (or very rarely).
but will not stop serious malware
And a shocking number of tutorials recommend bind-mounting docker.sock into the container without any warning (some even tell you to mount it "ro" -- which is even funnier since that does nothing). I have a HN comment from ~8 years ago complaining about this.
Imagine naming this executable "ls" or "echo" and someone having "." in their path (which is why you shouldn't): as long as you do "ls" in this directory, you've ran compromised code.
There are obviously other ways to get that executable to be run on the host, this just a simple example.
OTH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here.
The $HOME/.{aws,docker,claude,ssh}
Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.
Go and Rust tend to lend themselves to these more restrictive environments a bit better than other options.
Thanks for mentioning it - but now... how does one deal with this?
* but if you’re used to bind-mounting, they’ll be a hassle
Edit: This is by no means comprehensive, but I feel compelled to point it out specifically for some reason: remember not to mount .git writable, folks! Write access to .git is arbitrary code execution as whoever runs git.
You might still want to tighten things up. Just adding on the "rootless" part - running the container runtime as an unprivileged user on the host instead of root - you also want to run npm/node as unprivileged user inside the container. I still see many defaulting to running as root inside the container since that's the default of most images. OP touches on this.
For rootless podman, this will run as a user with your current uid and map ownership of mounts/volumes:
podman run -u$(id -u) --userns=keep-idThe only serious company that I'm aware of which doesn't understand that is Microsoft, and the reason I know that is because they've been embarrassed again and again by vulnerabilities that only exist because they run multitenant systems with only containers for isolation
Its all turtles, all the way down.
But for a typical case, VM's are the bare minimum to say you have a _secure_ isolation boundary because the attack surface is way smaller.
While I generally agree with the technical argument, I fail to see the threat model here. Is it that some external threat would have prior knowledge that an important target is in close proximity to a less hardened one? It doesn't seem viable to me for nation states to spend the expensive R&D to compromise hobbyist-adjacent services in a hope that they can discover more valuable data on the host hypervisor.
Once such expensive malware is deployed, there's a huge risk that all the R&D money is spent on potentially just reconnaissance.
systemctl --user enable podman.socket loginctl enable-linger <USER>
?Just a random thought, but if you can create a user on the host that just has the most minimal set of systemd services enabled your container needs, you could apply it to that user.
But still, on a server that wouldn't make much sense imho, as the default user is usually the service user having a minimal set of services enabled. On a desktop, your default user is logged in anyways. So I think this isn't a real problem tbh.
https://support.broadcom.com/web/ecx/support-content-notific...
https://nvd.nist.gov/vuln/detail/CVE-2019-5183
https://nvd.nist.gov/vuln/detail/CVE-2018-12130
https://nvd.nist.gov/vuln/detail/CVE-2018-2698
https://nvd.nist.gov/vuln/detail/CVE-2017-4936
In the end you need to configure it properly and pray there's no escape vulnerabilities. The same standard you applied to containers to say they're definitely never a security boundary. Seems like you're drawing some pretty arbitrary lines here.
I suggest people fuck around and find out, just limit your exposure. Spin up a VPS with nothing important, have fun, and delete it.
At some point we are all unqualified to use the internet and we used it anyway.
No one is going to die because your toy project got hacked and you are out $5 in credits, you probably learned a ton in the process.
As opposed to email?
If I accepted your version of the events then you need to accept you posted a link stating that spam makes up 45% of email traffic, 45% is neither half nor the majority. I suggest you actually read your "hard data" before posting it.
you have to be a fool to think there is less spam 2 years later but thank you for proving once again your clear bias and inability to read.