I agree that this new construct makes the landscape even more complicated, but I disagree that k8s has reached the point of over-engineering. Most of the parts of k8s are still essentially complex to me -- they're what you'd need if you wanted to build a robust resource pool management kind of platform.

Ironically, the push to "simplify" the platform with various add-on tools is what is making it seem more complicated. Rather than just bucking up and telling everyone to read the documentation, and understand the concepts they need to be productive, everyone keeps building random, uncoordinated things to "help", and newcomers become confused.

For example, I don't know who this operator framework is aimed at -- it's not at application developers, but at k8s component creators who write cluster-level tools, but what cluster tool writer would want to write a tool without understanding k8s at it's core? Those are the table stakes -- if I understand k8s and already understand the operator pattern (which is really just a Controller + a CRD, two essential bits of k8s), why would I use this framework?

I think if they really wanted to help, they'd produce some good documentation or a cookbook and maintain an annex of the state of the art in how to create/implement operators. But that's boring, and has no notoriety in it.

> You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering.

well consider you wanted to have a High Available solutions that supports Blue Green / Rolling Deploys without downtime. You either built it yourself or you rely on something like k8s. It's not that much over engineering. K8s is a lot of code, yes. But the constructs is still pretty simple. I think deploying k8s is still way easier than most other solutions out there, like all these PaaS's and Cloud solutions. Spinning up K8s is basically just using ignition/cloud-config, coreos and PXE or better iPXE. Yeah sometimes it's troublesome to upgrade a k8s version or a etcd cluster. However everything on top of k8s or even coreos itself is extremly simple to upgrade.

inb4 or our current system is using consul, haproxy, ansible and some custom built stuff to actually run our stuff. System upgrades are still done manually or trough ansible and my company plans to replace that with k8s. it's just way simpler to keep everything up-to date and run for high availability without disruption on deployments. it's also way simpler to actually get new services/tools into production, i.e. redis/elasticsearch without needing to keep them up to date/running.

It's not clear for me, if you confuse the compose/swarm development progress (which is like your ideals) and kubernetes (which afaik always was over-engineered to begin with).

Kubernetes has the huge day-1 problem, that it doesn't solve all of your problems. The hard stuff like networking and distributed storage are hook-in APIs. That's fine on Google's cloud where all the other stuff is there and was developed with these interfaces in mind, so all the endpoints are there. But most companies don't work in GCP/AWS alone. The moment you come on-premise you see that kubernetes only does 25% of what it needs to do to get the job done.

So, oyu have this tool who already lacks 75% in its original design and it tries to overcome this by adding more stuff. Then you combine this with a prematurely hyped community who just adds more stuff to solve problems that are solved, that don't need to get solved or that aren't problems, just to get their own names and logos into what's out there.

These two are patterns that make it very clear that it is impossible for Kubernetes to ever become a lean, developer friendly tool. But it's a great environment to make money already, I can tell you. And I think maybe that was the main goal from the beginning.

There's some truth and some wistful hope in your post; In my time at Google, the only thing that was anything like these "Operators" was what was developed by the MySQL SRE team, which was great but they also admitted it was a bit "round peg, square hole". There's a shared persistence layer that hasn't quite shown up yet; you need a low-latency POSIX filesystem and a throughput-heavy non-POSIX system (Chubby and GFS in the Borg world/Etcd and ??? in k8s). Not having the ability to work with persistent, shared objects is the biggest detriment to the ecosystem. S3 sorta works if you're in AWS, GCE supports Bigtable and etc

Operations is always more complex than folk expect it to be and product evolution typically reflects that. Kubernetes was simple because it couldn't miraculously teleport to do all the things ever-larger clusters require of it.

We forever rush to the limits of current technology and then blame the technology.

I think it's worth noting that Kubernetes never tried hard to impose an opinion about what belongs to the operator (as in the person running it) and what belongs to the developer. You get the box and then you work out amongst yourselves where to draw the value line.

Cloud Foundry, which came along earlier, took inspiration from Heroku and had a lot of folks of the convention-over-configuration school involved in its early days. The value line is explicitly drawn. It's the opinionated contract of `cf push` and the services API. That dev/ops contract allowed Cloud Foundry to evolve its container orchestration system through several generations of technology without developers having to know or care about the changes. From pre-Docker to post-Istio.

Disclosure: I work for Pivotal, we do Cloud Foundry stuff. But as it happens my current day job involves a lot of thinking about Kubernetes.

I just find docker swarm to be so much simpler and works for the type of stuff we deploy.

Sad to see it 'lose the race' against kubernetes.

Any plans to switch from templatized YAML to Jsonnet/Ksonnet?

And that’s why the team created a Helm-operator based on Lostromos (a tool for expanding Helm charts without running a tillers erver) that makes it easy to use a Helm chart for templating but still add additional lifecycle on top:

https://github.com/operator-framework/helm-app-operator-kit

The templating aspect of Helm and the set of quality content is complementary to being able to add higher level lifecycle.

>FWIW, I'm one of those who remember your name, simply because you pop up in every Kubernetes discussion with a predictably contrarian, long-winded opinion.

There are several k8s conversations on HN each day. I skip most of them.

> I don't know what you're getting out of it.

I get a lot of useful feedback out of it. Most of the posts I make on HN are about trying something out and gauging the response, because I want to learn from it. Sometimes I get a complete correction, which is great, because I stop believing something that's wrong. More often I get minor shifts in my personal POV, perspective on what arguments are effective and which aren't, the pitfalls/commmon counterarguments to specific positions, good feedback on tone / interpretation, and lots of other valuable information. Also, once in a while I make a nice personal connection.

I'll note for the record that response on k8s/containers is mixed. There are certainly a substantial number of threads where I'm at the bottom, but there are also threads where I make essentially the same arguments and score pretty well, along with a few supporting comments from people who say they don't get it either. HN's responses generally seem to be signaled by the tone of the thread and headline, and the preponderance of existing responses. If the groundwork is laid with a positive outlook, negative comments will usually have a hard time, and vice-versa.

Also, my arguments are usually not purely repetitive, even if they have the same core message (because same core things remain relevant). I have never talked about Operators on HN before. They came up and they're a good example of how people pretend that Kubernetes is more production-ready than it is, by obfuscating things like "you have to write special programs to teach Kubernetes how to deploy and manage your applications because the YAML configuration interface they tout isn't good for complex cases", behind the much hipper "Build a Kubernetes Operator, then you'll be cloudified and Dockerized out the wazoo!!"

I admit that I find a culture focused on this type of hype to be grating and immature, and as a sign of its inability to really bring substantial improvement to the table. I don't think I feel this way about anything new in general, I just think it's a reaction to a progressively-worsening engineering deficit in the "devops" field. I hope that I can learn whether this is right or wrong as time goes on.

> In this case, you're not wrong — but the curmudgeonly, somewhat tone deaf way that you go about it isn't very nice, which probably explains the downvotes.

Yeah, so this is a great example of why I continue to post about this. Most people would consider this subject matter very dry, but Kubernetes is something that people imbue with much more personal identification than is typical for infrastructure orchestration projects. Where are the CloudFoundry Diego disciples (paging jacques_chester ;) )?

It's important to post and learn the pressure points, and if there is any argument or circumvention that is effective against that identity imprint. I'm still trying to learn, so I continue to post and draw feedback from the community. I appreciate your participation in teaching me thus far. :)

First, I don't think anyone notices or cares when I post specifically, so it's hard to really feel like my "general opinion" is well-known enough to not talk about it anymore. I'm no Joel Spolsky over here!

Second, I feel like it's valid to point out that Operators are not really just a method of "packaging", in a post that tries to make it sound like Operators are just a small bit of YAML or metadata. You're writing real, non-trivial Go code that tells Kubernetes explicitly how to deploy and manage the lifecycles of specific types of applications.

At least until now with the "Operator Framework", there wasn't really even anything that firmly defined an Operator as an Operator; it's just what some people called their Go code that manipulated k8s's object handling and lifecycle internals.

But, if you insist, here's one operator I've worked with: https://github.com/coreos/prometheus-operator . This is from CoreOS itself.

Here's a patch I submitted about a year ago: https://github.com/coreos/prometheus-operator/pull/289 . This required updating the way the software handled HTTP response codes in one of its "watcher" daemons (because all packaging methods need those, right?), and fixing the order of operations in the bootstrap scripts.

Some more general info about this repo:

    $ du -sh prometheus-operator/.git
        51M     prometheus-operator/.git

The repo size is 51M.

    $ git rev-list --count master
        1716

There have been almost 2000 commits.

    $ cloc --vcs=git --exclude-dir vendor,example,contrib .
         290 text files.
         278 unique files.
         121 files ignored.
    
    github.com/AlDanial/cloc v 1.74  T=0.82 s (295.4 files/s, 48117.2 lines/s)
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    Go                              50           1581           1392          20622
    JSON                             9              0              0           6276
    YAML                           132            260            885           4164
    Markdown                        30            792              0           2957
    Bourne Shell                    17             65             58            257
    make                             1             34              2             91
    Python                           1             10              5             40
    TOML                             1             11             20             30
    Dockerfile                       2              8              0             27
    -------------------------------------------------------------------------------
    SUM:                           243           2761           2362          34464
    -------------------------------------------------------------------------------

It appears there are over 20k lines of Go code after excluding vendor libraries and the example and contrib directories (arguably, contrib should've been included).

I dunno, it just feels a little disingenuous, to me, to say that something that involves this much code is just a "packaging method" for a normal application. "Sure, just write an operator to package that up" like it's comparable to a package.json manifest or something. It's not! You need custom daemons that watch files to make sure that your k8s deployment stays in sync, and then you need to exert very meticulous and specific control over Kubernetes' behavior to make things work well.

I think it's demonstrative that it takes north of 20k lines of Go code to package an application for deployment on Kubernetes. What do you think?

-------------

EDIT: And one clarification: my opinion on containers as such is probably not well-known, since you're conflating it with my opinion on Kubernetes.

I like containers conceptually (who wouldn't?) and I run several of them through LXC:

    NAME               STATE   AUTOSTART GROUPS IPV4            IPV6 UNPRIVILEGED 
    axxxx-dev          STOPPED 0         -      -               -    false        
    gentoo-encoder     STOPPED 0         -      -               -    false        
    jeff-arch-base-lxc STOPPED 0         -      -               -    false        
    jeff-crypto        RUNNING 0         -      xxx.xxx.xx.xxx  -    false        
    jeff-ffmpeg        STOPPED 0         -      -               -    false        
    jeff-netsec        STOPPED 0         -      -               -    false        
    jeff-ocr           STOPPED 0         -      -               -    false        
    localtorrents-lxc  RUNNING 0         -      xxx.xxx.xx.xxx  -    false        
    nim-dev            STOPPED 0         -      -               -    false        
    plex-2018          RUNNING 1         -      xxx.xxx.xxx.xxx -    true         
    unifi              STOPPED 0         -      -               -    true    

I believe this is the kind of thing people actually want. Highly efficient, thin "VMs" that are easy to manage and run as independent systems without requiring the resource commitment.

There is a good place for Kubernetes in probably about 1% of deployments where it's used. Most other people are just trying to run something like LXC, but they're confused because everyone who is critical of k8s drops to -4 and gets HN's mods after them. :)

Minor correction: I've replied a couple of times on HN and gotten no response. You once clarified that this is because you don't check HN often enough to reply before the deadline, which is fine. I do admit that you once sent me an email and that I didn't reply to it.

cookiecaper covers his lack of response to k8s threads in his bio at https://news.ycombinator.com/user?id=cookiecaper. There is an email if you want to contact him directly.

I agree with your conclusions that the world is becoming more declarative and that we need to converge on an agnostic system to make these declarations.

However, you might've been trained incorrectly, perhaps as a joke, because you've assigned properties to the opposite solutions!

Configuration management like Ansible or Salt are agnostic modules that can be plumbed under the hood to work with any platform: use Ansible's "copy" module and it can be implemented under the hood as cp, rsync, passenger pigeon, whatever. You can use the same playbook against any target, including targets that run on orchestration platforms like Kubernetes.

Kubernetes Operators, on the other hand, are tightly linked to Kubernetes internals. They are not declarative; they require explicit instruction on how to manage your application's lifecycle, implemented in a statically-typed programming language. Indeed, if your system doesn't need to muck around in the Kubernetes internals, you don't create an "Operator", you just use the pre-baked object types like Service, Deployment, etc.

So I agree in principle, but Kubernetes is the opposite of what you've expressed here.