What job interviews taught me about Kubernetes(notnotp.com) |
What job interviews taught me about Kubernetes(notnotp.com) |
The uniformity is nice, we were moving from apps running directly ec2 instances provisioned with ansible. Each time we spun up a new service it was a process to get the ec2 instances provisioned just so.
But k8s is such a pain in the ass. One thing that I think people new to it don’t realize is that it’s not at all batteries included - to get a basic managed cluster setup, you’re still going to be installing a bunch of additional controllers (ingress, cert-manager, external dns to start). And then you’re on the hook for making sure all those processes stay up (hope the admission webhook controller for a critical resource doesn’t go down!). Then you’ve got to do a major upgrade on not only your cluster, but all of those controllers every ~3 months. And no one is shy about introducing breaking changes.
Also you’re introducing a huge amount of complexity with the k8s networking and dns layer that most startups have zero need for (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS).
I think there is a real hole in the market for a simple solution that lets you deploy some containers to some instances in a declarative fashion without all of that complexity and does decent LTS versions. I imagine there’s something out there that does this, but k8s has really sucked up all the oxygen.
Everyone initially wants thing A. But then they want to customize it to do all permutations and combinations n of A, B, C. They want it to be extensible. They want redundancy. They want orchestration. They want integration.
It’s why practically every config file format eventually becomes its own scripting language. Even HTML started off simple — now ridiculously complex — all the more ironic since practically nobody writes it by hand. Instead of CSS simplifying it, it became more complex.
There is another thing that is extremely customizable and extensible. It’s called a programming language. People write programs to solve specific problems.
There seems to be a perverse trend of cobbling together a Byzantine mesh of libraries, plugins, and services with complex configuration files to make it do practically everything possible. We just used to write software for such purposes…
And for anyone who thinks HTML is simple… the A (anchor) tag has an “ping” attribute that results in POST requests to a list of URLs when a link is clicked ! The list of attributes and resulting variations in behavior is quite mind boggling. It was supposed to be a damn link! https://html.spec.whatwg.org/multipage/links.html
https://github.com/openrundev/openrun is a project I am building. It supports declarative deployments, on a single-node with Docker or onto Kubernetes. The target use cases is limited to standalone web app, like internal tools. No support for stateful services, you manage stateful services yourself. With that simplification, OpenRun provides a much easier developer experience.
https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...
Perhaps those days are behind us.
And if you can do this again, what's your solution to reverse proxy, certificate management, DNS...etc? I guess you can docker-compose some custom stack on a single machine, maybe add one more machine then you can say it's HA enough for small scale. But you can also spend the same amount of time to install those kubernetes controllers with zero customization. In my experience, if you go with the default configuration, most of the well-maintained k8s components are boring as hell these days.
> (if you’re on EKS, make sure to read about scaling and monitoring CoreDNS)
If load to your service increases, you need to scale up/out your service. This is universally true. Do you have a proprietary solution that's easier and more reliable than bumping up the replicas count in kubernetes?
There are lots of design decisions in Kubernetes that I hate. But if you want me to choose between Kubernetes and any proprietary stack, in 2026, I would definitely choose Kubernetes.
I have a strong preference for renting bare metal and it has served me extremely well.
As for EKS, having to monitor and manually scale the built in DNS service or else my queries are just going to stop resolving is not the type of thing I expect to have to manage on a managed service. I see they have finally released autoscaling for CoreDNS, though it took them 6 years.
Hashicorp's Nomad basically is just that, supports various way of running stuff too which is neat. Shame about the license change which basically killed all my interest in it, so seems the hole is indeed still unfilled.
You can still add pods if needed and the systemd integration works.
Plus you can actually improve isolation by co-hosting services under separate UIDs.
Like any container it is just co-hosting, and elasticity is a bit slower with autoscaling instances, but it removes most of the complexity of K8s which very few org benefit from or have the culture to support.
Though as I recall, it makes heavy use of consul, which I have used in anger, and makes me a little weary (though that experience is likely very out of date).
Some self-reloading HAProxy in nomad to automatically assign URLs to services when needed. Could have used Consul but meh.
Tailscale for private networking.
CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point it to the external load balancer which I would have known if I read the documentation). trafeik is all you need. talosos largely mitigates the bare metal problems and comes pre-hardened and pre-optimized.
I solo most of my development projects and have used k3s for all of them. The only complaint is that cert-manager by default will fail silently and your certificates will expire. I largely mitigated this by having proper visibility setup via grafana and automated alerts (warns if certificates are about to expire) which should have been done by me anyway.
Two years ago I'd agree, today with LLMs everything I have runs talos with fully automated updates and I haven't had to be on-call for almost a year.
K8s is easier at smaller scales (I understand k3s as a packaged version ?), but you still need one or two people in your team to properly understands all of the concepts and inner workings of k8s, and be able to neck deep into if/when shit hits the fan.
For a small team that's a lot of commitment for something that is usually not their bread and butter and wish they could build once and only slightly tweak every year or so.
Just use ECS / Fargate with an ALB in front if you need a simpler use case.
Unless of course, all of the busywork that comes with kubernetes IS the value (to the engineer). Perhaps a bunch of engineers know at some level that locking the company into an overcomplicated cloud-within-a-cloud setup that has all sorts of weekly issues and requires constant work gives them a lot of job safety that they wouldn't get if they just used an AWS autoscaling group and you're done for the next 5 years.
Because simpler solutions DO exist (like a loadbalancer in front of an autoscale group, and not making a giant SOA for an app that orders you taxis, or books you a bnb or whatever nonsense).
I mean, it's CDK and whatever equivalents other providers have, isn't it? If you fully embrace all the stuff they give you then it's straightforward to declare everything and it all works together. The downside is the vendor lock-in but unless you actively deploy to multiple environments, which most people don't, you're probably locked in in various ways without knowing about it.
The problem is that when you run this long enough you want K8s features anyway.
Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes.
I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.
There's a certain type of engineer (maybe 25% of them) who does "hype-driven-development." No matter the technology, they are huge advocates for the technology. The hype may be absolutely real, complete nonsense (e.g. mongodb), or somewhere in between (ai). The vast majority of the time it's hype for a new technology that feels 90% the same from the end-user perspective (react vs vue, docker vs colima, go vs other, whatever vs whatever).
These engineers though, only care about something when it's new and trendy enough to be a differentiator. This is because they don't give any hoots about the actual usefulness of anything, they are just trying to differentiate themselves in a market by leveraging vibes rather than raw competence. I think these types of engineer drove kubernetes for companies that don't need it, but tipped the scales enough that it has critical mass.
The irony being kubernetes is way too heavy/clumsy an abstraction for most companies. The savings of packing pods onto the same node is usually a tiny fraction of the engineers' salaries who are managing it.
The other irony is now that kubernetes isn't the new sexy thing, but a standard tool that AI or a normie can do all the hard work for, the hype driven engineers are off looking for the next thing.
And I do think there is a way to use kubernetes with minimal damage, but it requires making firm rules about not focusing on things that aren't needed yet (e.g. istio) and making firm hiring choices about only people who understand that such optimizations are complete wastes of time for a series A startup.
Ask your favorite GPT to generate manifests, get primary app into cluster with telepresence or execute straight from container and switch contexts and clusters like it's 90s again.
One reason I dislike Docker Compose and Docker is lack of isolation. Yes sure if you put your arm deep enough you can get it, but on local k8s I can spin cluster per workspace and not worry about conflicting ports between PostgreSQL instances.
Before LLMs writing consistent YAMLs was PITA but today on low/development scale it's pretty much free lunch.
Now am laid off, and hard to find a job...
Using Kubernetes because you're unable to grok docker's networking enough so you can't run multiple containers using their own ports and not conflicting with other stuff sounds like a recipe for disaster, even (especially?) if you use agents for this. Particularly if you let them manage a production environment, you're bound to lose important data eventually.
> pretty much free lunch.
Aah, famous last words of the young :)
But I found funny that the OP summarized to use Kubernetes when CTO is no longer the only dev.
I ended up in a different non-SRE role but if you're interested in working on it, please let me know and I'd love to walk you through it.
To use it is a whole different question, and not in any way related to job interviews. I have worked in places that are crazy for not using it and others where using it was even crazier.
That the tech benefits may not be there, but they’re using it for the non-tech benefits
> My personal threshold would be the moment the CTO isn't the only engineer anymore. As soon as a second person shows up, the problems K8s solves become real.
Unrelated to the content of the article, this sentence structure is a dead giveaway of LLM writing.
Pretty much, almost. Have spent a bunch of time in my career working on the "VM + systemd" setups, stuff running on a rack, or in an ec2 on cloud - managed kubernetes is a lot better for me than those cobbled together messes. There's "easier" setups but usually end up costing me a lot more in time and $.
To answer simply, it became good + convenient. I could complain about plenty, and people here like to, but honestly you couldn't pay me to go back to the old way. The one legitimate gripe is the upgrade schedule is exhausting, on AWS it's about every 6 months before you go into extended support. I also hate being at the mercy of arbitrary decisions like "ok we know a huge chunk of the web going back a decade has architected off our Ingress API, but recently we decided we dont really like that way anymore and we want you to use Gateway API instead, so, um, like ya we know it just killed off one of the most used open source ingress configs (ingress-nginx) but yea trust us bro this is going to be so much better" kind of thing.
It is nice to be able to have a consistent deployment pattern, with traceability, rollback support, and production approval checks. It’s nice to not have some archaic something stuck in someone’s head. It’s also nice to be able to see how something works by reading the code, which is usually up to date and deployable.
I’d like to gently push back on that. ;-D
Terraform, when committed to git, provides organisational memory. But less so uniformity, since all providers are different (and you should expect different things when applying). No tracing besides git. And tfstate is hard to share between developers, unlike kube state.
Kubernetes is more the same across providers. And it manages drift after something is applied, which is not a direct argument of OP, but a strong reason over other IAC.
And yes, I also enjoy how well deploying works. And how things generally fit together. Liking the networking complexity less so.
Really? For years and years we put our tfstate files into private S3 buckets at $DAYJOB and it seemed to work just fine. We didn't even take pains to ensure that everyone was on the very same version of the Terraform CLI. What problems did you guys run into?
My current company makes this claim, but it's not true. They also have serverless apps, and also have some services running directly on EC2.
They just think of the Kubernetes deployments as the "standard" way.
> Second was shared, hireable knowledge. K8s is basically a lingua franca now.
People were demanding experience with Kubernetes, long before it was reasonable to expect it. Everyone added it to their resume, because they had to.
I worked once at a bankm fully kubernetes, the amount of problems were out of reality from this world.
Complexities are being added for no reason at all.
K8s is a complicated beast. CTOs hiring for their 10 person company because of its "used everywhere" is a bad reason to adopt a major piece of technology. You can always graduate to it later if need be.
It removes the overhead of a lot of what sysadmins and devs of yesteryear did by hand or had to have a career's worth of experience to do quickly.
That's not to say that people don't need to know what they're getting into when they adopt kubernetes but especially when you're using a managed offering and not on the bleeding edge of what it supports it's pretty easy in terms of overhead and maintenance.
Right now, I’m one dinosaur managing a startup’s tech portfolio. Everything lives in my head first, then in my break-glass vault for addressing the bus problem. Our public cloud footprint is a single KMS for backups. We have no VMs, everything is a cloud service.
The literal fucking second we have real infrastructure requirements for compute, it’s right to GCE. No ifs, ands, or buts. Here’s our Git Repo, here’s the managed K8s control plane, make it work.
If (or when) we need on-prem compute, we add them to the K8s control plane as worker nodes and taint accordingly.
It’s just so much more interchangeable, even if the learning curve for non-SDEs can be a little steeper than VMs.
I think what you hear is never the whole story, there is much more going on.
Their identified reasons are OK though.
I'll admit I'm dreading switching over to the gateway api, but by the time I get forced off ingresses it should be a stable/mature ecosystem. That's still a ways out though.
I don't know anyone still dealing with VMs anymore, except our IT guy who manages a couple of pet servers for random executives from the before times. In the last year k8s has started absorbing executive pet processes and the number of VMs our IT guy manages has dropped by about half.
While I'm here spouting stuff, yeah hiring for k8s is real easy, if our SRE gets hit by a bus, he can be replaced in a week, and we can probably struggle through using opus until that happens. K8s being he lingua franca of git ops IaC makes it real easy for the new guy to parachute in and start working. Every VM thing is going to be totally bespoke and have the personality of the guy who designed it, which is rarely a good thing.
Even on AWS EKS, you will run into bullshit with their network overlay. Egress policies are a mess (at least half a year ago, you were not able to say something like "allow pod A to egress traffic to service (!) B" despite a service resolving down to an IP address in the end.
And that's before going into the unholy mess that is getting connectivity to and from the external world to your cluster. Cloudfront, ACM certificates, ALB, ALB-EKS integration, Route53, Route53-EKS integration, EFS, EFS-EKS integration, EBS, EBS-EKS integration, RDS, RDS-EKS integration, IAM-EKS integration, SSM, SSM-EKS integration, autoscaling... and if you want more pain and don't already wince, try setting that up across regions or, as I had to do once, across account boundaries.
Kubernetes is powerful. But do not make the mistake of assuming it's easy to get started with, at least on the admin side. Even if you got prior AWS experience, getting it all integrated into EKS so you don't have to deal with Terraform and helm/k8s for a full deployment of a piece of software will take you an awful lot of time.
For users though? It's a breeze, I will admit as much. Everything down to the firewall rules can be encoded in k8s spec files.
Oh it's not necessary per se but if you want to host a web service with any sort of state and not having to do stuff in parallel either by hand or by terraform, I'd consider the integrations pretty vital.
It's easy enough (well, it's still addons whose versions you have to keep updated each on their own) once it is set up, but getting to the point where you have something reproducibly running for the first time is annoying as hell.
> do stuff in parallel either by hand or by terraform
…specifically by terraform. Making k8s own the provisioning and management of external infrastructure on principle (as opposed to when that makes sense, e.g. load balancers/gateway/CSI providers) is not a good approach. Sure, it feels unified, but the cost of unification is incredibly not worth it.