Increasing Attacker Cost Using Immutable Infrastructure

Increasing Attacker Cost Using Immutable Infrastructure(diogomonica.com)

110 points by diogomonicapt 9 years ago | 36 comments

xorcist 9 years ago |

It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage.

The article tries to pitch read only Docker images some kind of solution, but running your applications read only (and what other permissions you grant your application) has nothing to do with Docker images. Using file system and process namespaces for application isolation is a good idea.

But lately it seems to be getting more popular to drop untrusted applications in containers (possibly even under outside control and full of who-knows-what) as if that somehow solves "security". The thing is, you still need to be able to reason about what permissions your respective application requires, there's no getting away from that.

feld 9 years ago | |

The invention of containers (FreeBSD jails) was to automate restoring vulnerable web apps that couldn't be patched / updated (edit: as well as allowing shared hosting with different perl requirements to co-exist on the same server) Every time tampering was detected they automated restoration and watched the attacker squirm as their changes kept reverting and files were disappearing.

Don't dismiss this concept. It's a perfectly valid approach in some scenarios.

drieddust 9 years ago | |

> It is not a good idea to restore attacker-owned applications to a "known good" state before you have done at least a cursory post mortem. Not only do are the security holes intact but since the attacker now knows they been found out, you can invite more serious damage.

I would also add that attackers are actually after the data. Exploiting application vulnerabilities is just a mean to that end so bringing back exploitable application from the previous image is a BAD idea.

Having said that previous image can be a good starting point to patching up the vulnerabilities and bringing the application online.

ris 9 years ago | | |

> I would also add that attackers are actually after the data.

Not strictly true - depending on the parties it's still a very desirable goal to be able to snoop on a site's continued operations, inject some code into visitors' pages, or just have a platform for further attacks.

wepple 9 years ago | |

I agree that rolling back an image to a "known good" state is not the appropriate solution; if you can do it efficiently, what's stopping you from doing it very regularly?

Attacking a system that disappears and restarts on a regular basis is a nightmare for current attackers; it's not something that they have the tooling to deal with yet.

I believe some of these ideas were discussed by Dino Dai Zovi in a talk[1] he did which combined a whole bunch of rather out of the box ideas on defense.

[1] https://www.youtube.com/watch?v=_A6Y62AcQ2g

hodgesrm 9 years ago | |

I'm not sure I understand the argument against reverting state. If you are cleaning up any sort of mess--including getting hacked--it's vastly easier to start with a good before-things-got-messed up state. This is one of the reasons people buy expensive storage arrays that support fast snapshot reversion.

Also, the author is arguing fix now but preserve all information for fault diagnosis later. For a lot of problems that seems like exactly the right choice.

Senji 9 years ago | |

Should probably just tiger-tree-hash your containers. If they deviate you give them a copy of your database, but it's a "virtual" shadow copy. No writes to it reflect on the real database.

toomuchtodo 9 years ago | | |

If attackers get your database, you lose. Containers don't fix application vulnerability mitigation and leaking data.

thinkmassive 9 years ago |

I know the author is pitching the immutable infrastructure side, which definitely has its merits. But the ability to so easily diff the image and running state of a container also opens some intriguing opportunities for honeypot automation.

Are there any open source solutions that already take advantage of features like this? Or are those mostly kept secret for security and business reasons at this time?

zokier 9 years ago | |

Intriguing indeed. Something like when a write happens, instead of just plain blocking it redirect the write to a honeypot environment and send out an alert. Sure, that won't fool a good attacker for a very long time, but would be interesting to capture various drive-by and skriptkiddie attacks.

runeks 9 years ago |

    > Until we fix this RCE vulnerability, the attacker will 
    > still be able to execute code on our host [...]

With Docker, it seems to me like we're moving closer and closer to the server being an executable of its own, but with the necessary Linux kernel bits compiled in such that it can execute on (virtualized) hardware.

I'm wondering how far we can take this. The ability to execute code on the host is there because that's what Linux does, but what if we removed this interface, and replaced it with an interface that just accepts one or more ELF binaries at compile-time? Then these would become the only Linux executables that this kernel can execute.

As far as I can see, we could do the same to system calls: if an executable can enumerate all the system calls it needs, we can compile a kernel that will accept only these system calls, which should be a small subset of all available Linux syscalls.

JoachimSchipper 9 years ago |

Docker is not (designed to be) a security technology. Yes, rolling back servers/VMs/containers to their previous state is a good capability to have (although most of us just use backups for that!), but assuming that an attacker cannot break out of a container is, at least, optimistic.

wepple 9 years ago | |

> assuming that an attacker cannot break out of a container is, at least, optimistic.

Agreed. However as long as you don't look at it as your primary line of defense, it increases the cost to an attacker. And that's currently the best we can ever do.

JoachimSchipper 9 years ago | | |

There are arguments - which are at least plausible - that the same effort is better spent on VM / Solaris Zones / FreeBSD jails / traditional chroot / SELinux / just using dedicated hardware for everything / ...; several of these have the advantage of not encouraging people to throw up a compartment (of some kind) and never update it again, which is certainly something that happens with Docker.

sublimino 9 years ago |

Of note is that an immutable/noexec filesystem doesn't prevent code being downloaded to an environment var/typed out and run - tools like https://github.com/SafeBreach-Labs/pwndsh just pipe source to an interpreter (in that case BASH, which generally isn't installed in smaller base images).

Reducing the attack surface is important, but if a running container is compromised it's imperative a post-mortem is performed immediately - and the issue remediated - to prevent re-exploitation.

justincormack 9 years ago | |

potentially you do not need any interpreters available at all, which certainly increases attack difficulty.

axelfontaine 9 years ago |

This is definitely one of the main reason we decided to this road for VMs as well at Boxfuse (https://boxfuse.com): minimal images (5 MB for a full VM with a Go application) and immutable infrastructure all the way.

oolongCat 9 years ago |

So I am confused, how does any of this increase attacker cost? Cant you do the same thing at the OS level already? Wouldn't making the dir read only do the same thing?

If your data gets stolen and your website defaced what's the use of immutability? I mean you can always run a diff tool against the current code on the server with the code you have on your repo right?

viraptor 9 years ago | |

> how does any of this increase attacker cost?

Because it forces the attacker to write a specific payload for your service. Standard, reused "drop shell.php and register IP" will not work anymore. And realistically if the target of the attack was a WordPress installation, it will likely be a trivial, automated script.

> Cant you do the same thing at the OS level already?

Yes, you can. Even better, split execution privileges from file privileges, then make it read only, then put a grsec/apparmor/selinux profile on the service. It's not docker specific, but docker does make read only service a little bit easier.

> Wouldn't making the dir read only do the same thing?

Yeah, but who would do that old school thing. Docker security! :-(

dgoldstein0 9 years ago | |

it's pretty hard to diff filesystems that aren't designed for it. Either you need to lock the whole filesystem somehow - e.g. by taking it offline - or you have to deal with the fact that other processes are reading/writing as you scan the filesystem, which is rather difficult to reason about.

And it's not just about diffing your code with your repo - that only works if the attacker tried to attack your code. What about other running processes? New files on the system containing malicious code, outside of the paths you usually deploy code to? what about new, unexpected cron jobs?

Overall, it could become a pretty complex job. A filesystem with some intrinsic snapshotting makes this a lot easier.

Senji 9 years ago | | |

COW by default file systems which provide a "snapshot at time x" can help with this. NTFS can do it with shadowcopy.

icebraining 9 years ago | | |

You can also use lvm snapshots, which work with any filesystem.

Animats 9 years ago |

Real immutability would be good in more applications, as when you load a program into the memory of an embedded processor and then blow the programming fuse so it can never be changed. Hardware where parts of memory can be made read-only after boot and can't be written again without a full processor reboot would be useful.

bureado 9 years ago |

I remember when mounting /tmp noexec and chmod g-w on htdocs was enough.