Defence Against the Docker Arts(blog.heroku.com) |
Defence Against the Docker Arts(blog.heroku.com) |
Honestly after years of stagnation, the most exciting work on container building is now coming out of Docker. Buildkit is amazing, a real hidden gem.
Implementing CNBs as a buildkit frontend would break key security and performance features. For instance, CNBs can build images in unprivileged containers without any extra capabilities, which buildkit cannot do. CNBs can also patch images by manipulating their manifests directly on a remote Docker registry. This means that image rebuilds in a fresh VM or container can reuse layers from a previous build without downloading them (just metadata about them), and base images can be patched for many images simultaneously with near-zero data transfer (as long as a copy of the new base image is available on the registry). As far as I know, buildkit can't do any of that yet.
That said, we do plan on using buildkit (once it ships with Docker by default) to optimize the CNB pack CLI when you build images without publishing them to a Docker registry. It's a huge improvement over the current Docker daemon implementation for sure!
It sounds like CNB will break compatibility with the massive Dockerfile ecosystem, in exchange for... sometimes not downloading a layer? That is not appealing to me at all, because Dockerfiles are too embedded in my workflow, losing support for them is simply not an option.
As for unprivileged builds, I don’t see any reason buildkit can’t support it since it’s based on containerd.
I think it’s a mistake not to jump on the buildkit/docker-build bandwagon. You would get a 10x larger ecosystem overnight, basically for free. Instead it seems like you’re betting on CNB as a way to “kill” Dockerfiles. But users don’t actually want to kill anything, they want their stuff to continue working. Without a good interop story, you’re pretty much guaranteeing that CNB will not get traction outside of the Pivotal ecosystem. Seems like a shame to me.
There's a good introduction here: https://medium.com/@tonistiigi/build-secrets-and-ssh-forward...
I think the main problem is trying to apply the same patterns to stateful and stateless services.
Thanks for posting that!
I'd note that the frontend there is for "v2b" buildpacks -- the previous generation of Cloud Foundry buildpacks. The CNB lifecycle has changed a fair amount from the v2a (Heroku) and v2b designs.
FROM node:10 AS build
WORKDIR /foo
COPY . .
RUN npm i
RUN npx webpack
FROM nginx:whatever
COPY nginx.conf /etc/nginx/config.d/
COPY --from=build /foo/dist /srv
...
The other thing I'd like to see is the ability to output multiple containers from one Dockerfile. There is so much wasted work where I have webpack stuff and a go application that run in separate containers but are built together. There is one Dockerfile like the above to build the static part. There is another to build the go binary and copy it to an almost-pristine alpine container (including dumb-init, cacerts, tzdata, and grpc_health_probe). I don't understand why I have to have two Docker files to do that.
It's one readable text file used to recreate an entire environment. It's sort of a picture worth a thousand command lines.
That said, I wish there was a way to get rid of all the && stuff, which is used to avoild writing a layer of the filesytem.
Why not have something like:
RUN foo
RUN bar
RUN bletch
LAYERAnd squashing comes with a major drawback: you lose layer caching and any hope of a vaguely-efficient rebuilding process.
cf push
also, if you're just a user of docker you might not care about dockerfiles. (like most people building software don't care about the Makefiles and prefer not to look at them)
You could have the same issue by simply trying to rpmbuild your app. No really, you are just doing packaging. If you want more comfort, look into how redhat or Debian maintain their packages. They have similar problems and most likely they have mature solutions.
E.g. look at phusion/baseimage or phusion/passenger - they are production ready and with good defaults. But you also have an easy way to augument them with latest ffmepg compiled from sources to support some exotic format for video conversion worker.
Basically, why do the work yourself? Especially if someone else will solve the weird problems and keep everything up to date with no effort on your part.
I know from personal experience that buildpacks maintainers have seen stuff you people wouldn't believe. Attack ships on fire off the shoulder of -02. gcc beams glittering in the dark near the Nokogiri gate.
Some key points:
- CNBs can manipulate images directly on Docker registries without re-downloading layers from previous builds. The CNB tooling does this by remotely re-writing image manifests and re-uploading only layers that need to change (regardless of their order).
- CNB doesn't require a Docker daemon or `docker build` if it runs on a container platform like k8s or k8s+knative. The local-workstation CLI (pack) just uses Docker because it needs local Linux containers on macos/windows.
The OCI image format expresses layer order as an array of digests. Essentially, "read the blobs with these SHAs in this order, please".
Cloud Native Buildpacks have predictable layouts and layering. A buildpack can know that layer `sha256:abcdef123` contains (say) a .node_modules directory. It can decide to update only that layer, without invalidating any other layer.
And the operation can be very fast, because you can do it directly against the registry. GET a small JSON file, make an edit, POST it back.
This is a big deal because under the classic Dockerfile model, changes in a lower layer invalidate the higher layers. But this means your image can be invalidated by OS layer changes, dependency changes and so on. It's the right policy for Docker to have -- a conservative policy -- but Buildpacks have the advantage of additional context that lets them rely on other guarantees. Most noticeably ABI guarantees.
If you want to scale in a pinch then it's a case of making some tiny tweaks and pushing to Heroku instead.
Not 100% sure if that is the fault of the dokku implementation or buildpacks in general though. It's all IO related, so I might not even have noticed it had I had faster disks.
I'm not too fussed as I only deploy once in a blue moon if I need to quickly update my website or something. I can see it getting frustrating if you've got a larger product/app doing CI and deploys every merge to master.
Although there are some issues with buildpacks in production, they are a great set of training wheels. I'm excited about what's happening with the next generation of buildpacks, CNB is a standard with the support of more than just Heroku – Pivotal is "the other buildpacks company," also involved, and I'm not sure how many other companies!
It's clear sometimes that when you work with containers at scale, that what you're building a lot of layer cakes, and you need to treat the layers separately to remain effective at scale. Some of the ideas like swapping layers out are pretty far out and cool.
Opinionated Platforms Are Risky: The CloudFoundry platform was more opinionated than some competing platforms in the market. In fact, the biggest debate between CloudFoundry and its direct competitors was about whether customers need opinionated platforms or not. CloudFoundry only supported 12 factor applications whereas platforms built on top of Kubernetes could support both stateful and stateless (12 factor) applications.
If you're building a stateless 12-factor app and there's a buildpack that does what you want, buildpacks are clearly better than lower-level Dockerfiles. But there's no buildpack for something like a database and there probably never will be, so the flexibility of directly building containers needs to exist.
That's why Cloud Foundry pioneered the Open Service Broker API.
For what it's worth, that article's recounting of history has substantial variations from my own recollection of history.
edit: Also, clicking to the article, the actual title seems (now?) to be: "Turn Your Code into Docker Images with Cloud Native Buildpacks", so I'm even more confused now... o_O
Yes, because for end-user developers, you don't need any files. The files you mentioned are used by buildpack authors.
Buildpacks fit nicely into the heroku way of doing things. But at any medium to large sized engineering organization, there's no way they could satisfy the requirements for even a simple majority of services that using a Dockerfile provides.
Disclosure: I've worked on Cloud Foundry Buildpacks twice and worked on this latest effort until recently.
Yeah, throw that code over to the ops team. Let them figure stuff out themselves.
Generally speaking I don’t know of many dev teams that enjoy OS and middleware patching or the nuances of Linux security. Some do of course, but many just want to code.
Buildpacks’ vision is that the ops team already knows how to install run production systems and that knowledge is encoded and tweaked in a buildpack.
This is a solution in search of a problem. Please stop.
Right now for one of our standard large customers, remediating a critical CVE may take several hours, whether they are using current-generation buildpacks or Dockerfiles and build farms. CNBs will drive that figure down to minutes. Thousands of distinct applications, dozens of sites, several tens of thousands of containers, billions of requests per day, patched a few minutes after the buildpack releases from automation observing hundreds of distinct upstream dependencies across a dozen language ecosystems.
Anyone with enough money and people and patience can build and maintain this kind of a capability for themselves. But it's a lot cheaper and easier to pay someone else to maintain it for you.
Though buildpacks differ from most of these by skipping Dockerfiles altogether.
Assuming most people arnt going to change thier whole platform because they didnt bother rtfm, or googling.
Yes docker doesnt handle this well, but there is a lot more nuance to that example.
-4 votes btw.
There are so many morons out there trumpeting the buzzword, and they are just total imbeciles. My last pointy-haired boss thought he could manipulate the conversation to discover what I know about containerizing EC2 instances, by guiding the conversation with leading questions that'd get me to spill my guts with nerd signaling and posturing, so that he could rip off whatever we discussed and create competition among all his direct reports.
He did this with everyone that worked for him. He'd get us alone in a one-on-one, frame the conversation as a casual discussion where we pick each others brains and sketch out flow charts and relational UML diagrams on a whiteboard, but sprinkled on top, he'd throw a peer under the bus, sell them out, and ask how I might do it better.
On some level, some of my peers really were rotten bastards; complete shit heels; lazy, arrogant pogues. But if pointy-haired boss is doing it to them, he's doing it to me too. I'm just a rube off the street, an ass in a chair at 9 AM on the dot, a warm body with a pulse after all. Replacable and modular as a docker container, yes, yes?
So docker as a buzzword gave this boss a boner. A disgusting, throbbing, pulsating, glistening, dog-dick-red, boss boner. He wanted it sucked, and the way to suck it was to containerize bullshit with docker. So I started telling him the lowest effort shit, and planned my escape, because the guy was toxic waste, and I didn't feel like being a toadie.
It's like jesus, anyone who knows what docker is, knows it to be little more than a glorification of shell scripts and tar files. It doesn't do anything revolutionary. It's a plastic milk crate instead of a cardboard box. If you need to pack your widget with styrofoam peanuts, well, that's your problem.
Pointy-haired boss thought that docker was an art form, like package design is to product unboxing vlogs. He wanted to relish and savor the moment of unboxing the most expensive Apple product ever unboxed before the eyes of youtube.
Sorry boss man, spend the rest of your life with the lifers you can't fire. I'm done with your deceptive, hype gobbling office persona.
https://medium.com/@tonistiigi/build-secrets-and-ssh-forward...
the link you've supplied regards ssh access when running docker images, not when building them.
... i did notice that for lots of downloads, it is somewhat slower, or perhaps more prone to lag, than authentication from inside the running image.
Flick and swoosh.
And devs like fast builds too :)
I also don't really see how this project is "incompatible" with anything in the docker ecosystem. I hope Dockerfiles have a long and healthy life. CNBs are another method to build OCI compatible images that provides a lot of benefits for some types of users and use cases.
I expect that the pack cli will eventually build on top of and take advantage of Buildkit
You’re right that more flexible patching and optimizing transfers are valuable. But those problems are independent of build frontends: you could solve them once for both buildpacks and Dockerfiles. In fact buildkit is well on its way to doing exactly that.
Basically I would prefer if buildpacks and Dockerfiles could all be built with the same tooling. CNB seems like a wasted opportunity to do that, because it bundles two things that should be orthogonal: a new build format, and a new build implementation. Docker is going in the opposite direction by unbundling the format (Dockerfile) from the implementation (buildkit).
You can build an image with both technologies, but that's not the key to the argument. The key here is every Dockerfile is unique and potentially quite different from any other Dockerfile. Small differences in layer order and layer contents multiply to very large inefficiencies at scale.
The way you tackle this problem is to make the ordering and contents of layers predictable for any given software that is being built. You can achieve this with Dockerfiles with golden images, strict control of access to Dockerhub, complicated `FROM` hierarchies, the whole shebang.
But at that point you are reinventing buildpacks, at your own expense.
Note that this doesn't change with or without buildkit.
> a new build format, and a new build implementation. Docker is going in the opposite direction by unbundling the format (Dockerfile) from the implementation (buildkit).
Is your understanding that we invented a new image format or that we rewrote most of Docker? Or that the way we've written it prevents, for all times and all purposes, adopting buildkit as part of the system in future?
Because both of those are misapprehensions. We have extensively reused code and APIs from Docker, especially the registry API.
If you like Dockerfiles, you like Dockerfiles. Lots of people do. I did, until I'd used them for a while.
I'm not sure what you mean by "break compatibility". CNBs produce OCI images. They'll run on containerd just fine.
As for ecosystem: you'll note that the domain is heroku.com.
Can you provide an example of how layer order can cause an issue?
FROM nodejs
COPY /mycode /app
RUN npm install /app
An NPM buildpack can signal that there's nothing to change, allowing the overall lifecycle to skip re-building that layer.
There's also the problem of efficient composition. Suppose I have this:
RUN wget https://example.com/popular-shell-script.sh && \
go get https://git.example.com/something-else@abc123 && \
./popular-shell-script.sh && \
rm ./popular-shell-script
RUN go get https://git.example.com/something-else@abc123
But I don't get to do that, because the layers have different overall contents and different digests. Buildpacks don't get you all the way to a file-centric approach, but because they follow a repeatable, controlled pattern of selecting the contents and order of layers, they greatly improve layer reuse between many images.
Part of what we do in Cloud Native Buildpacks is to use the layer abstraction more aggressively to make image updating more efficient. That requires taking care with the ordering and contents of each layer, so that they can be replaced individually.
Putting it another way: we don't see the image as the unit of work and distribution. We're focused on layers as the central concept.
If you just untar the layers on top of each other, foo will still be there. This is a problem, no?