Learning containers from the bottom up(iximiuz.com) |
Learning containers from the bottom up(iximiuz.com) |
I disagree with this:
> Now, when you have a decent understanding of containers - from both the implementation and usage standpoints - it's time to tell you the truth. Containers aren't Linux processes!
This is a bit of wordplay, I'm assuming, in absence of a word that defines the operating system features that power the concept of containers. To Linux, there is no (to my knowledge) concept of a "container". The container runtime runs your process(es) as the parent and uses the operating systems features to isolate it and restrict it/them. A virtual machine would just be a full emulated version of this, rather than using the operating system to virtualize the network stack. The author is right in that there is no such thing as a container, but only as much as containing is a thing you do, imo. What users think of containers are still just processes though, and I don't think that's an entirely useless abstraction to be cognizant of.
Under the hood, that's all containers are!
If running the same properly designed software exhibits material differences in behavior between a bare metal process and a containerized process, then they aren’t the same as a matter of practical semantics. Ironically, virtualized processes have much closer equivalence to a bare metal process than containerized processes in practice.
Saying a container is “just a process” is like saying a virtual machine is “just a process”, both are true in some sense depending on how you define “process”. But as a matter of practical engineering, they are different kinds of things.
Fantastic distillation. Thank you!
Fortunately the container architecture is flexible so that you can use as much or as little of it as you like.
I also tend to think that if you want stronger isolation for security purposes then you will want a lightweight VM rather than a container (and if you are worried about side channels, probably hardware partitioning - good luck.)
Incorrect. The images are mere files(/subtrees), and you can write one however you wish.
In particular it breaks software that does I/O scheduling in user space, which is idiomatic and explicitly supported by the Linux kernel, even on virtual machines, but this use case conflicts with the container abstraction so runtimes offer an ersatz version that allows the code to run albeit poorly.
What syscalls do you think are intercepted, how? Speaking as someone who can write kernel code, I'm not aware of any such thing specific to containers. (As far as the linux kernel is concerned, there's no such thing as a container.)
If you're talking about BPF, that can be used outside of containers, e.g. systemd can limit any unit, and using it is not part of a definition of what a container is.
You can find many examples in the wild of reputable software that loses significant performance once containerized no matter how configured. Literally no one has demonstrated state-of-the-art data infrastructure software that works around this phenomenon, and at this point you’d think someone would be able to if it was trivially possible. I test database kernels in a diverse set of environments and currently popular containers aren’t remotely competitive with VMs, never mind bare metal. The reasons for the performance loss are actually pretty well understood at a technical level, albeit esoteric.
Every popular container system has runtimes that intercept syscalls. Whether or not Linux requires containers to intercept syscalls is immaterial because in practice they all do in a manner destructive to I/O performance.
There used to be a similar phenomenon with virtual machines for many years, such that no one deployed databases on them. Then clever people learned how to trick the VM into letting them punch a hole through the hypervisor, and we’ve been using that trick ever since. It isn’t as fast as bare metal, but it is usually within 10%. No such trick exists for containers and as a consequence performance in containers is quite poor.
Can you say more on this trick. Is it available by default on the cloud VMs, or is it something that has to done in the user's code. I have seen tweets from DirectIO developers saying AWS Nitro machines are better for kernel-bypass-IO compared the Google cloud VMs. but my understanding of it was something done by Amazon Nitro card engineers/developers, and Google was working on something similar to improve the performance.
All high-performance data infrastructure bypasses the operating system for most things, taking total control of the physical hardware it uses. Cores, memory, storage, network. Linux happily grants that control, with caveats. It doesn’t work the same inside Kubernetes and Docker, and no one can figure out how to turn it off. Maybe you don’t work on I/O intensive applications, but for people that do this is a well-documented phenomenon. My teams have wasted far too many hours trying to coax good performance out of code in containers that works just fine on bare metal and virtual machines.
[0] https://www.nginx.com/blog/comparing-nginx-performance-bare-...