It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch.
We had to fork BuildKit very early in our Depot journey. We've fixed a ton of things in it that we hit for our use case. Some of them we tried to upstream early on, but only for it to die on the vine for one reason or another.
Today, our container builders are our own version of BuildKit, so we maintain 100% compatibility with the ecosystem. But our implementation is greatly simplified. I hope someday we can open-source that implementation to give back and show what is possible with these ideas applied at scale.
This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are already ossified.
It's really rare to see these sorts of projects decomposed into building blocks even just having code organization that helps a newcomer understand. Despite all the code being out in public, all the important reasoning about why certain things are the way they are is trapped inside a few dev's heads.
Managers and executives are happy to hear that you made the builds faster or more reliable, so the infra people who care about this kind of thing don't waste time on design docs and instead focus on getting to a minimum prototype that demonstrates those improved metrics. Once you have that, then there's buy-in and the project is made official... but by then the bones have already been set in place, so design documentation ends up focused on the more visible stuff like user interface, storage formats, etc.
OTOH, bazel (as blaze) was a very intentionally designed second system at Google, and buildx/buildkit is similarly a rewrite of the container builder for Docker, so both of them should have been pretty free of accidental engineering in their early phases.
This is a very insightful comment
It's clear a ton of care has gone into the product, and I also appreciated you personally jumping onto some of my support tickets when I was just getting things off the ground.
But the real hidden power of buildkit is the ability to swap out the Dockerfile parser. If you want to see that in action, look at this Dockerfile (yes, that's yaml) used for one of their hardened images: https://github.com/docker-hardened-images/catalog/blob/main/...
BuildKit is very cool tech, but painful to run at volume
Fun gotchya in BuildKit direct versus Dockerfiles, is the map iteration you loaded those ENV vars into consistent? No, that's why your cache keeps getting busted. You can't do this in the linear Dockerfile
Everything runs in its own docker runner. New buildkitd service for every job. Caching only via buildkit native cache export. Output format oci image compressed with zstd. Works pretty great so far, same or faster builds and we now create multi arch images. All on rootless runners by the way
Had to recently make it so multiple versions can run on the same host, such that as developers change branches, which may be on different IaC'd versions (we launch on demand), we don't break LTS release branches.
What nobody tells you is that the cache mount is local to the builder daemon. If you're running builds on ephemeral CI instances, those caches are gone every build and you're back to square one. The registry cache backend exists to solve this but it adds enough complexity that most teams give up and just eat the slow builds.
The other underrated BuildKit feature is the ssh mount. Being able to forward your SSH agent into a build step without baking keys into layers is the kind of thing that should have been in Docker from day one. The number of production images I've seen with SSH keys accidentally left in intermediate layers is genuinely concerning.
It's about not having the private keys stored unknowingly in intermediate layers of a build container.
Let me have side effects, I'm a consenting adult and understand the consequences!!!
It sounds great in theory, but it JustDoesn'tWork(tm).
Its caching is plain broken, and the overhead of transmitting the entire build state to the remote computer every time is just busywork for most cases. I switched to Podman+buildah as a result, because it uses the previous dead simple Docker layered build system.
If you don't believe me, try to make caching work on Github with multi-stage images. Just have a base image and a couple of other images produced from it and try to use the GHA cache to minimize the amount of pulled data.
I find that buildah is sort of unbearably slow when using dockerfiles...
It's yet one more incomprehensible Buildkit decision. The original Docker builder had a very simple cache system: it computed the layer hash and then checked the registry for its presence. Simple content-addressable caching.
Buildkit can NOT do this. Instead, it uses a single image as a dumping ground for the caches. If you have two builders using the same image, they'll step on each other's toes. GHA at least side-steps this.
But I tried the registry cache, and it didn't improve anything. So far, I was not able to get caching to work with multi-stage builds at all. There are open issues for that, dating back to 2020.
The code first options are quite good these days, but you can get so far with make & other legacy tooling. Docker feels like a company looking to sell enterprise software first and foremost, not move the industry standard forward
great article tho!
Docker is profoundly bad software.
Edit: The claim about the hash being the same is incorrect, but an identical Dockerfile can produce different outputs on different machines/days whereas nix will always produce the same output for a given input.
The cache key includes the state of the filesystem so I don’t think that would ever be true.
Regardless, the purpose of the tool is to generate [layer] images to be reused, exactly to avoid the pitfalls of reproducible builds, isn’t it? In the context of the article, what makes builds reproducible is the shared cache.
If they didn't take shortcuts. I don't know if it's been fixed, but at one point Vuze in nix pulled in an arbitrary jar file from a URL. I had to dig through it because the jar had been updated at some point but not the nix config and it was failing at an odd place.
Also, there is nothing stopping you from creating a container that has make + tools required to compile your source code, writing a dockerfile that uses those tools to produce the output and leave it on the file system. Why that approach? Less friction for compiling since I find most make users have more pet build servers then cattle or making modifications can have a lot of friction due to conflicts.
https://tuananh.net/img/buildkit-llb.png
Maybe the page was changed? If you're just talking about the gaps between lines, that's just the line height in whatever source was used to render the image, which doesn't say much about AI either way.
Compared to a Dockerfile it's just too hard to follow
A build system is meant to yield artifacts, run tools, parallelize things, calculate dependencies, download packages, and more. And these are all things that have some algorithmic similarity which is a kind of superficial similarity in that the failure modes and the exact systems involved are often dramatically different. I don't know that you can build something that is that all-encompassing without compromising somewhere.
Blaze is/was integrated with Omega scheduler, which is not open.
Bazel is open source.
By the time Bazel was open sourced, Twitter had pantsbuild and Facebook had buck.
OpenWRT's Makefiles are sufficient to build OpenWRT and the kernel for it. (GNU Make is still sufficient to build the Linux kernel today, in 2026.)
Make compares files to determine whether to rebuild them if they already exist; by comparing file modification time (mtime) unless the task name is in the .PHONY: list at the top of the Makefile. But the task names may not contain slashes or spaces.
`docker build` and so also BuildKit archive the build chroot after each build step that modifies the filesystem (RUN, ADD, COPY) as a cacheable layer identified by a hash of its content.
Other Dockerfile instructions add metadata: CMD, ENTRYPOINT, LABEL, ENV, ARG, WORKDIR, USER, EXPOSE <port/tcp>, VOLUME <path>.
The FROM instruction creates a build stage from scratch or from a different container layer.
Dockerfile added support for Multi-stage builds with multiple `FROM` instructions in 2017 (versions 17.05, 17.06CE).
`docker build` is now moby and there is also buildkit? `podman buildx` seems to work.
nerdctl supports a number of features that have not been merged back to docker or to podman.
> it obviously was designed for a system where most dependencies are vendored, and worked better for languages that google used like c++, java, and python.
Those were the primary languages at google at the time. And then also to build software? Make, shell scripts, python, that Makefile calls git which calls perl so perl has to be installed, etc.
Also gtests and gflags.
"Compiler Options Hardening Guide for C and C++" https://news.ycombinator.com/item?id=43551959 :
>> There are default gcc and/or clang compiler flags in distros' default build tools; e.g. `make` specifies additional default compiler flags (that e.g. cmake, ninja, gn, or bazel/buck/pants may not also specify for you).
Which CPU microarchitectures and flags are supported?
ld.so --help | grep "supported"
cat /proc/cpuinfo | grep -E '^(flags|bugs)'`
How to add an `-march=x86-64-v3` argument to every build?
How to add build flags to everything for something like x86-64-v4?
Which distros support consistent build parametrization to make adding global compiler build flags for multiple compilers?
- Gentoo USE flags
- rebuild a distro and commit to building the core and updates and testing and rawhide with your own compiler flags and package signatures and host mirrored package repos
- Intel Clear Linux was cancelled.
- CachyOS (x86-64-v3, x86-64-v4, Zen4)
- conda-forge?
Gentoo:
- ChromiumOS was built on gentoo and ebuild IIRC
- emerge app-portage/cpuid2cpuflags, CPU_FLAGS_X86=, specify -march=native for C/[C++] and also target-cpu=native for Rust in /etc/portage/make.conf
- "Gentoo x86-64-v3 binary packages available" (2024) https://news.ycombinator.com/item?id=39250609
Google, Facebook, and Twitter have a monorepo to build packages from.
Google had a monorepo at the time that blaze was written.
Twitter ("X") is moving from pantsbuild to blaze BUILD files.
TIL there is a buck2. How does facebook/buck2 compare to google/bazel (compare to what is known about blaze)?
Should I build containers (chroot fs archives) with ansible? Then there is no buildkit.
FWIW `podman-kube-play` can run some kubernetes yaml.
- install ansible in-band and run it against localhost (sucks because your playbook is in a final image layer; you might not want Python at all in the container)
- use packer with ansible as your provisioner and a docker container export, see: https://alex.dzyoba.com/blog/packer-for-docker/
- copy a previous stage's root into a subdirectory and then run ansible on that as a chroot, afterward copy the result back to a scratch container's root.
All of these options fall down when you're doing anything long-running though, because they can't work incrementally. As soon as you call ansible (or any other tool), then from Docker's point of view it's now a single step. This is really unfortunate because a Dockerfile is basically just shell invocations, and ansible gives a more structured and declarative-ish way to do shell type things.
I have wondered if a system like Dagger might be able to do a better job with this, basically break up the playbook programmatically into single task sub-playbooks and call each one in its own Dagger task/layer. This would allow ansible to retain most of its benefits while not being as hamstrung by the semantics of the caller. And it would be particularly nice for the case where the container is ultimately being exported to a machine image because then if you've defined everything in ansible you have a built-in story for freshening that deployed system later as the playbook evolves.
Source: I was on part of the team that did the open sourcing, and had to clean lots of cruft in the code base that had accreted over 8 years.
I’m afraid you’re missing my point, though. A high quality build system takes fixed inputs and produces outputs that are, to the extent possible, only a function of the inputs. If there’s a separate process that downloads the inputs (and preferably makes sure they are bitwise identical to what is expected), fine, but that step should be strictly outside the inputs to the actual thing that produces the release artifact. Think of it as:
artifact = build_process(inputs)
inputs = fetch(credentials, cache, hashes, etc)
inputs = …
assert hash(inputs) == expected
Once you have commingled it so that it looks like:
final output, intermediate layers = monolithic_mess(credentials, cache, etc)
Docker build is not a good build system, and it strongly encourages users to do this the wrong way, and there are many, many things wrong with it, and only one of those things is that the intermediate layers that you might think of as a cache are also exposed as part of the output.
Hence my confusion of what you meant -- no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people do it wrong, which is bad.
We're talking about the same thing basically. Yes fully defining your inputs to the container by passing in the keys is a good solution.
> the container is also a build artifact
By "build artifact" I mean the data that is the output of the build and get distributed to other machines (or run locally perhaps). So a build artifact can be a tarball, an OCI image [0], etc. But calling a container a build artifact is really quite strange. A "container" is generally taken to mean the thing you might see in the output of 'docker container ls' or similar -- they're a whole pile of state including a filesystem, a bunch of volume mounts, and some running processes if they're not stopped. You don't distribute containers to other machines [1].
> in context of CI, the output of running the build using the container
I have no idea what you mean. What container? CI doesn't necessarily involve containers at all.
> no one's saying ssh keys are in the CI build artifacts. But obviously they can be in the container as layers if people do it wrong, which is bad.
If the build artifact is an image, and the keys are in the image, then the keys are in the build artifact.
> Yes fully defining your inputs to the container by passing in the keys is a good solution.
Are you suggesting doing a build by an incantation like:
$ docker run --rm -v /input:[sources] -v "/keys:($HOME)/.ssh" my_builder:latest /input/build_my_thing
Yes, I realize that almost everyone fudges this because we have lots of tools that make it easy. Even really modern stuff like uv does this.
$ uv build
[0] https://specs.opencontainers.org/image-spec/
[1] A build artifact could be a container snapshot, but that's different.
Flakes, then, run in a pure evaluation mode, meaning you don't have access to stuff like the system triple, the current time, or env vars and all fetching functions require a hash.
similarly, nix cannot guarantee reproducibility if the user does things to break that possibility
Nix isn't some magical thing that makes things reproducible either. nix is simply pinning build inputs and relying on caches. nixpkgs is entirely git based so you end up pinning the entire package tree.
There's always:
RUN dnf remove -y ansible && dnf clean all
ansible/ansible-builder: https://github.com/ansible/ansible-builder
"How to Use ansible-builder to Create Execution Environments" https://oneuptime.com/blog/post/2026-02-21-how-to-use-ansibl... :
> The Build Process Explained: When you run ansible-builder build, it goes through these steps:
> Reads your `execution-environment.yml` definition, Resolves collection dependencies (including transitive dependencies), Generates a `Containerfile` in a `context/` directory, Copies dependency files into the build context, Runs the container build using Podman or Docker
How to Use Ansible to Build Podman Images > Building with Build Arguments, Multi-Stage Builds, Building with Buildah: https://oneuptime.com/#building-with-buildah :
It's possible to build images using build a commands in an ansible playbook:
buildah from
buildah copy
buildah run
buildah config
buildah commit
The CLI way to traverse an ansible playbook graph, for e.g. dagger:
ansible-playbook --list-tags
ansible-playbook --tags {tagname}
It probably shouldn't (?) parallelize because that wouldn't be a deterministic build; installing A then B is not the same as installing B then A. (Is not the same thing as installing A in one container image layer, B in another container image layer, and then trying to merge the package databases.) A given package B could conditionally install or configure according to whether or not A is already installed, and so for example package install tasks are not commutative.
.
Bootc (osbuild) builds VM and native machine images from Containerfiles:
bootc-image-builder
bootc upgrade
"Demonstrate a debian or arch base image" bootc-dev/bootc#865 https://github.com/bootc-dev/bootc/issues/865
Is packer necessary with bootc?
To require signatures for containers and also for native containers with bootc:
cat /etc/containers/policy.json | grep sigstoreSigned
podman image trust show --raw
I mean using the CI system to pass in keys or creds. Yes, it's better to build the image with dependencies, but sometimes you can't do that.
Docker has a `--no-cache` flag, even easier than blowing it away, which you can also do with several built in commands or a rm -rf /var/lib/docker
Perhaps worth revisiting: https://docs.docker.com/build/cache/