Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
So duh.
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
How?
Because that is the thing, not only it is a YAML spaghetti, everything changes depending on the puzzle pieces.
I had to follow CNCF related podcasts only to be aware of what cool projects were changing Kubernetes all the time.
Thankfully nowadays I only care about managed containers, regardless how the hyperscalers do it, it is no longer my headache.
Never needed to deal with vSphere directly.
I'm hardly trying to pretend like I'm a hyperscaler: I usually run k3s on a single node, and most of the time I admin it "clickops" style with k9s, something I find much easier than most other management tools.