Hi HN! I’m excited to share Holos, a Go command line tool we wrote to fill the configuration management gap in Kubernetes. Holos uses CUE to configure software distributed with Helm and Kustomize using a well defined, type safe language eliminating the need to template YAML. You probably know (or are) someone who has suffered with the complexity of plain text YAML templates and merging multiple values.yaml files together to configure software running in Kubernetes. We built Holos so we don’t have to template YAML but we can still integrate software distributed with Helm and Kustomize holistically into one unified configuration.

At the start of the pandemic I was migrating our platform to Kubernetes from virtual machines managed by Puppet. My primary goal was to build an observability system similar to what we had when we managed Puppet at Twitter prior to the acquisition. I started building the observability system with the official prometheus community charts [1], but quickly ran into issues where the individual charts didn’t work with each other. I was frustrated with how difficult it was to configure these charts. They weren’t well integrated, so I switched to the kube-prometheus-stack [2] umbrella chart which attempts to solve this integration problem.

The umbrella chart got us further but we quickly ran into operational challenges. Upgrading the chart introduced breaking changes we couldn’t see until they were applied, causing incidents. We needed to manage secrets securely so we mixed in ExternalSecrets with many of the charts. We decided to handle these customizations by implementing the rendered manifests pattern [3] using scripts in our CI pipeline.

These CI scripts got us further, but we found them costly to maintain. We needed to be careful to execute them with the same context they were executed in CI. We realized we were reinventing tools to manage a hierarchy of helm values.yaml files to inject into multiple charts.

We saw the value in the rendered manifests pattern but could not find an agreed upon implementation. I’d been thinking about the comments from the Why are we templating YAML? [4][5] posts and wondering what an answer to this question would look like, so I built a Go command line tool to implement the pattern as a data pipeline. We still didn’t have a good way to handle the data values. We were still templating YAML which didn’t catch errors early enough. It was too easy to render invalid resources Kubernetes rejected.

I searched for a solution to manage and merge helm values. A few HN comments mentioned CUE [6], and an engineer we worked with at Twitter used CUE to configure Envoy at scale, so I gave it a try. I quickly appreciated how CUE provides both strong type checking and validation of constraints, unifies all configuration data, and provides clarity into where values originate from.

Take a look at Holos if you’re looking to implement the rendered manifests pattern or can’t shake that feeling it should be easier to integrate third party software into Kubernetes like we felt. We recently overhauled our docs to be easier to get started and work locally on your device.

In the future we’re planning to use Holos much like Debian uses APT, to integrate open source software into a holistic k8s distribution.

[1]: <https://github.com/prometheus-community/helm-charts>

[2]: <https://github.com/prometheus-community/helm-charts/tree/mai...>

[3]: <https://akuity.io/blog/the-rendered-manifests-pattern>

[4]: Why are we templating YAML? (2019) - <https://news.ycombinator.com/item?id=19108787>

[5]: Why are we templating YAML? (2024) - <https://news.ycombinator.com/item?id=39101828>

[6]: <https://cuelang.org/>

user:	eloip
created:	December 5, 2023
karma:	5