Show HN: Holos – QEMU/KVM with a compose-style YAML, GPUs and health checks

Show HN: Holos – QEMU/KVM with a compose-style YAML, GPUs and health checks(github.com)

56 points by zeroecco 27 days ago | 23 comments

I got tired of libvirt XML and Vagrant's Ruby/reload dance for single-host VM stacks, so I built a compose-style runtime directly on QEMU/KVM.

What's there: GPU passthrough as a first-class primitive (VFIO, OVMF, per-instance EFI vars), healthchecks that gate depends_on over SSH, socket-multicast L2 between VMs with no root and no bridge config, cloud-init wired through the YAML, Dockerfile support for provisioning.

What it's not: Kubernetes. No clustering, no live migration, no control plane. Single host. Prototype, but I'm running it on real hardware. Curious what breaks for people.

imiric 27 days ago |

Very cool, thanks for sharing.

I built something similar recently on top of Incus via Pulumi. I also wanted to avoid libvirt's mountain of XML, and Incus is essentially a lightweight and friendlier interface to QEMU, with some nice QoL features. I'm quite happy with it, though the manifest format is not as fleshed out as what you have here.

What's nice about Pulumi is that I can use the Incus Terraform provider from a number of languages saner than HCL. I went with Python, since I also wanted to expose a unified approach to provisioning, which Pyinfra handles well. This allows me to keep the manifest simple, while having the flexibility to expose any underlying resource. I think it's a solid approach, though I still want to polish it a bit before making a public release.

khimaros 27 days ago | |

check out https://github.com/lnussbaum/incant

imiric 27 days ago | | |

Ah, that's neat as well.

I took a slightly different approach in that I don't want to use YAML as the authoritative source. Many projects abuse it, and end up creating a DSL on top of it with all sorts of hacks to achieve the flexibility of a programming language. Pulumi and Pyinfra already provide user-friendly primitives and idempotent(ish) APIs that work much better than YAML. I simply want to expose some (opinionated) building blocks to make them easy to use, and allow users to customize them and add their own as needed. E.g. I definitely don't want to write any shell scripts inside YAML. :)

BTW, Pulumi already supports YAML[1], which can be used with any provider. But to me it's too verbose and generic, and of course, it lacks the provisioning primitives.

[1]: https://www.pulumi.com/docs/iac/languages-sdks/yaml/

zeroecco 27 days ago | |

Your path makes a ton of sense too! You get typed languages, state management, and the Incus team's work on the QEMU layer. The tradeoff I wasn't willing to make is the daemon + state store: Incus wants to own the VM lifecycle the way libvirtd does, and once you have that you're back to "two sources of truth" if you ever shell out. Holos is deliberately stateless on the host; Everything lives under one directory per project, rm -rf is a valid uninstall (though it will abandon the VMs if running). Different answer to the same frustration. Would genuinely like to see your thing when it's public.

d3Xt3r 27 days ago | | |

This is exactly what I dislike about incus. But what I do like about incus is how I can easily spin up and configure VMs directly using the CLI, without preparing a config first (I hate yaml).

So would be nice if holos could replicate that docker/incus CLI functionality, like say "holos run -d --name db ubuntu:noble bash -c blah".

imiric 27 days ago | | |

> Incus wants to own the VM lifecycle the way libvirtd does, and once you have that you're back to "two sources of truth" if you ever shell out.

That's true. But I didn't want to reinvent what Incus or any hypervisor abstraction does. I simply wanted to add some sugar on top that allows me to easily declare infra using small abstractions, and to tie in the provisioning aspect along the way. I still use Incus directly, and can benefit from their work, as you say. State is also managed by Pulumi, so really, there are 3 places for it to exist. There are some challenges with this, of course, but I think the tradeoff is worth it.

Good luck with your project, I'll be keeping an eye on it. I'll probably make a Show HN post when I release mine. Cheers!

frabonacci 27 days ago |

it reminds me of https://github.com/dockur/windows with its compose-style YAML over QEMU/KVM. The difference i'm seeing is scope: dockur ships curated OS images (Windows/macOS), while holos looks more like a generic single-host VM runner. Is that a fair read? also curious any plans to support running unattended processes for OS installs?

zeroecco 27 days ago | |

True and decent read. Holos is a generic runner underneath, and I'm not trying to compete on the curated-OS-image front; the assumption is you bring a cloud image and cloud-init does the rest. Unattended installs for Linux work today through cloud-init (that's most of what the YAML drives). Windows/macOS would need a different path.. autounattend.xml injection for Windows, custom payload for macOS; I haven't built that. If there's interest, a provisioner: type block that picks the right unattend mechanism based on OS isn't crazy, but it's not on the near-term list.

ranger_danger 27 days ago |

My biggest issue with QEMU is trying to keep up with seemingly arbitrary changes to the command-line options format over time, and keeping up scripts that can launch an image across different versions for testing.

Originally if you wanted e.g. a SoundBlaster16 device you could use -device sb16. Then it changed to -soundhw sb16. And now it's -audio driver=none,model=sb16; this has been happening with several different classes of options over the years and I haven't found any good documentation of all the differences in one place. If anyone knows, I'd appreciate it.

spidermonkey23 27 days ago |

I like it! is there a way to use both virt-manager and virsh with this or do we have to pick one. I'd be keen to migrate my VMs to this compose based system if it was a simple import job.

zeroecco 27 days ago | |

Thanks! Honest answer: nope. Holos talks to QEMU directly and skips libvirt entirely — that's deliberate (no XML, no daemon, state lives in a single directory per project), but I like the idea of being able to import vm's. I will try to find some time to add that feature.

tiernano 27 days ago |

Interesting. Wonder could it be modified to work with proxmox?

gmuslera 27 days ago | |

With Proxmox it could be as well Docker compose for LXC.

johnny22 27 days ago |

sure has been awhile since i thought about vagrant

zeroecco 27 days ago | |

Fair — the YAML shape is similar. The differences that matter to me: GPU passthrough works without fighting the provider plugin ecosystem, healthchecks gate depends_on instead of you sleeping in a provisioner, and there's no Ruby, no vagrant reload, no box format.

yjftsjthsd-h 27 days ago | | |

Honestly, if you just sold it as vagrant but with all the weird stuff fixed and actually made to play nice with the rest of the stack that I'd actually want to use, that would be a pretty compelling package to me

tadfisher 27 days ago | | |

Stop generating comments, please.

la64710 27 days ago |

What is it using to create the vm without libvirt?

spwa4 27 days ago | |

See https://github.com/zeroecco/holos/blob/5f5e60f753c5e99c421db...

Qemu command-line calls

khimaros 27 days ago |

another player in this space built on incus: https://github.com/lnussbaum/incant