As Kubernetes grows, a startup ecosystem develops in its wake(techcrunch.com) |
As Kubernetes grows, a startup ecosystem develops in its wake(techcrunch.com) |
What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. We're currently on Google Kubernetes Engine, and I'm disappointed in the lack of integration throughout Cloud Platform. GCP is great in many areas, but the Kubernetes offering is clearly an afterthought. As an example, if you create an ingress, this creates a bunch of load balancer objects (forwarding rules, URL maps, backends). But these are all given obscure names (k8s-um-production-zippediboop-f8da5e6c92f38300), and none of the Kubernetes labels transfer over. Same with disks; GCE disks don't even have labels, so they put a bunch of JSON in the "description", which of course cannot be filtered or queried. Similar things happen with VMs and other things; the "mapping" of Kubernetes to GCP concepts is basically either lossy or non-existent. Many other similar weaknesses exist: Cloud SQL (if you want to use Cloud SQL, you have to manually run the Cloud SQL Proxy as a sidecar), GCP service accounts (different from Kubernetes service accounts!), etc. GKE is solid, but everything seems like it's been put together without any over-arching design; GKE is just another thing buried in a somewhat ramshackle jungle of different services.
There's an opportunity for a vendor to come in and offer a platform where you start with Kubernetes. In particular, this means that the platform should be manageable from Kubernetes, through the use of CRDs and pluggable controllers. For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever).
That said, it's hard to compete with GCP at this point. Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.
I wouldn't be so sure about that. Consulting companies are drawn to operational complexity, of which Kubernetes has a lot of, like bears are to honey.
https://kubernetes.io/partners/
https://www.google.com/search?q=kubernetes+consulting&ie=utf...
Thing is, can you point to alternatives which are simpler? I cannot think of any, unless you only have a handful of containers to manage.
Even things like AWS's own ECS become difficult to manage as you grow.
To me this is just another hype train, like TensorFlow/Caffe before it, and like Hadoop/Spark before that.
These tools all have their uses in specific, non-common cases, but for your average business they are a net loss in time and money.
In my experience building and running Kubernetes clusters for Fortune 500 companies is that actually it's much simpler, it's just different from traditional infrastructure.
Different != Complex.
After years of using all the major clouds - Google Cloud has the best performance and primitives, but the worst support and operational/business features. If you want the fastest VMs/storage/networking, and are fine with non-standard APIs and beta SDKs, then it's a great platform and has super simple billing.
AWS and Azure are much better suited for most companies and startups though in terms of actually getting your business running with standard tooling, large ecosystems, and a massive portfolio of services.
What do you mean? You can do exactly that today, with persistent volume claims. K8s will create the storage volume for you and make it available for containers. This is "dynamic provisioning".
https://kubernetes.io/docs/concepts/storage/persistent-volum...
What is missing is being able to create other resources. For instance, define a EC2 instance as a k8s object and have K8s provision it, for legacy or anything you haven't moved to K8s yet(databases, etc). I see no technical reason why it cannot be implemented today.
We are basically bringing all the stuff we learned building Cloud Foundry to bear on Kubernetes. There's also an increasing amount of systems being shared. Right now Istio is seen as the long-term north-south and east-west routing solution for Cloud Foundry App Runtime (CFAR, what folks traditionally thought of as Cloud Foundry). We're looking at introducing Loggregator to the Kubernetes world as Oratos, we have factored buildpacks into standalone systems that can have a single image builder for any OCI-compliant platform, we led the formation of the Open Service Broker API effort that creates a common service catalogue approach for both CFAR and Kubernetes, we've contributed to runC and I lose track of all the rest.
When I say "we" I should say this isn't just Pivotal. IBM, SAP and SUSE are deeply involved as Cloud Foundry community members.
Disclaimer: I work on CSE and use it for K8s dev/test. It's really handy.
IBM Cloud Private is a Kubernetes-based private cloud offering, though from your other points I don't think it will have everything you want.
If the latter, I am not so sure (or else I don't understand your point.) A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those.
More subtly Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model. The best you can do is make the layering relatively efficient.
Databases run fine on Kubernetes and have been doing fine since 1.7, and the meme that Docker is bad for stateful apps is a getting a bit old. The challenges are mostly the same as with running databases on a VM or bare metal. In particular, you need to know how to manage HA.
The weakest point is perhaps that Kubernetes's scheduler completely ignores disk I/O, so you have to bare careful to avoid sharing a node with multiple disk-heavy apps that will compete for resources. This is includes the file system cache; for example, PostgreSQL works best when it can rely on the OS to cache pages, so you don't want other apps (including Docker and Kubernetes themselves) to compete there.
That said, I wasn't saying that a hosted solution shouldn't also offer VMs. Just that Kubernetes should be the main entrypoint and control plane. And someone figured out a way to run VMs via Kubernetes [1], which is a neat example of a solution to the lack of integration I was complaining about earlier.
[1] https://www.mirantis.com/blog/virtlet-run-vms-as-kubernetes-...
Not saying I've tried it, but some people are having ok luck pinning RDBMS pods in Kubernetes to specific nodes. It goes a bit against k8s principles, but it makes operational sense.
Kubernetes is baking in windows container support, for some legacy scenarios. More interestingly (and posted on HNs front page today), are solutions that allow independent VMs to be run as though they were k8s pods. This provides hybrid models ideal for legacy packaging and maintenance while moving onto new hardware.
> Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model
There will always be some natural impedance between hardware and software...
I think this picture is ripe for improvements though, and we're already seeing the edges of it take shape :)
Kubernetes will be moving towards smarter networking solutions to handle more and better use-cases with better performance (BPF), and is incorporating better network abstractions slowly but surely.
From the under layers: software defined networking (SDN), and kubernetes providers for major virtualization platforms (VMware, for example), have commercial offerings that could readily support integrated or "hyperconverged" operations... Microsegmentation and microservices go hand-in-hand. Empowered by a platform connected with role based access controls top-to-bottom there's a lot of potential to harmonize those distributed firewalling/VLAN needs through the same declarative YAML.
We're not there, yet. We are closer than ever though :)
Frankly, it was a supremely painful platform to work on. They obfuscated just enough of the k8s API to make it simultaneously completely unintuitive for less "orchestration minded" team members, yet severely underpowered for me and my fellow platform workers. It struck an unhappy medium for our team that no-one wanted or needed.
All my looks at OpenShift suggest that not enough has been peeled back to make it a useful platform-on-a-platform, but since they probably paved the way for certain features, those features have predominantly been implemented (and more tightly integrated) into k8s. RedHat is going to need to come up with a new value proposition for OpenShift for it to ever be a truly viable alternative to "raw" Kuberenetes, and given their buy-in to the technology, I'm not convinced they'll want to throw away the work they've done so far. Good money after bad, or something.
Not that Google is better here, but I already know what GCP provides; something that wants to compete with GCP/GKE really needs to explain why they're a viable competitor.
(And to people who design these things: If you need to have a "Products" dropdown filled with unexplained product names, you're doing something wrong.)
Also there’s already work being done on supporting cluster management from Kubernetes itself: https://github.com/kubernetes-sigs/cluster-api
As someone who is feeling burned over picking swarm over k8s, I'm not super into the idea of trusting a 3rd party to do k8s first.
Anything anyone implements well will be ported to gke in due time, no?
Operationalizing K8s can be fraught with vendor specific parts. It doesn't help documentation wise when many things gloss over a concept that is offloaded into GCE or GKE.
According to googlers here on HN, Google does use Kubernetes internally via GCP/GKE. But clearly those apps are in the minority, given Google's huge investment in Borg.
Google's spring-cleaning has nothing to do with their open source projects. Kubernetes is one of their most widely used open source "products" and the industry is creating an incredible community around it with several big players that could easily jump in to take over project leadership and financing.
It is actually a good thing that some k8s founding members left Google because they already got what they wanted to achieve and went on to work on stuff that Google probably wouldn't have prioritized as much, see Ark and kssonnet.
Kubernetes is IMHO the platform for cloud native applications for the next 5+ years.
Not wanting to use it because Google dropped everyone's darling (Reader) is rather naive.
I wouldn't equate Google's consumer facing products with their cloud/enterprise products.
It's kind of interesting because Google and Microsoft have opened a new front against Amazon to counter their strategy of locking you in with services. If software from CNCF is good enough to be used in place of these services on each cloud provider then you can nullify AWS's advantage there and Google can lure you in with ML and Microsoft with their enterprise experience.
But I'm not sure these projects can make progress faster than AWS can release and update their fantastic array of services. Kubernetes being the obvious exception (when is EKS GA?!?!)
Just want to remind anyone who hasn't tried K8. You can run MiniKube locally right on your laptop to get a taste of its power (and complexity):
https://kubernetes.io/docs/getting-started-guides/minikube/
For example, a 3-node Redis Enterprise cluster, all run locally:
https://redislabs.com/blog/local-kubernetes-development-usin...
Currently using GCloud and Stackdriver monitoring, but a few of the tools I am excited about include:
Prometheus / Granafa / KSonnet
KubeFlow ML
https://www.youtube.com/watch?v=I6iMznIYwM8
Istio, for programmatic routing
And, Agones, for game hosting
Recently, I spun up a simple pod-to-pod communication example but I found it pretty difficult. If you look up cluster networking in Kubernetes (https://kubernetes.io/docs/concepts/cluster-administration/n...) you'll find a whole fire hose of different options from ingress to calico to fabric and on and on.
This was what it took for me to try and rubber ducky my way to getting networking to work on Kubernetes, and in the end I had to get help from a friend at work (https://stackoverflow.com/questions/50195896/how-do-i-get-on...). It may be better than what came before, but it's not great.
After a few days of playing I set up lets encrypt with load balancing, a running app (Rails). A remaining issue are persistent volumes and how truly persistent these are. I haven't found out yet which solution I should pick for this. Longhorn is a Rancher product, which is probably what I will read more about now, but I cannot be sure. There are so many concepts and terminology that you need to figure out. Having Rancher in between is not helping me get a hang of kubernetes itself more purely of course.
A while back I was playing with Docker Swarm and I must say that I like Docker Swarm better in the sense that it feels closer to the source and because it is built into Docker. I get a feeling however that Kubernetes is where the future is so learning more about Docker Swarm is probably less worthwhile.
For those of us not backed by venture capital and not charging SV ex-googler rates to our clients, we need something to say "we'll host you on this git push containerization thing, it'll be cheap and easy, and we'll expand when the time comes, and that'll be cheap and easy too".
For $36/month (8GB Ram, 2vCPU) you get a single node if you just want to have the Kubernetes APIs. Maybe introducing smaller node plans later in the year.
Unfortunately, not much info on our website but please sign up if you are interested.
I did a (admittedly short) search for kubernetes and kernel bypass, and the only thing that seem remotely relevant was [0], however it didn't indicate whether they work together.
For background, I work for a Dark Pool Alternative Trading System, and we currently utilize kernel bypass for all of our networking using Solarflare NICs & openonload [1].
In the same vein, curious how containers work with CPU shielding and pinning threads to specific cores. Is it possible, and how do multiple containers on the same box interact in that regard. Do they need to be quasi-aware of each other so as to not pin a thread to the same core?
I'd greatly appreciate if anyone with experience with containers can answer these questions. I'm genuinely curious, but it's not worth researching further if there's no solution that can handle these strict requirements (e.g. it's a non-starter if containers increase latency).
[0] https://thenewstack.io/life-post-container-world/ [1] http://openonload.org/
At Datica (where I work), we started in 2014 with a bespoke container orchestration layer. This powered our HITRUST certified Platform. Think Heroku for regulated industries (like Healthcare). After years of hardship trying to keep up with the market we finally decided to go all in on K8s.
Kubernetes gives us the flexibility and community to focus on the compliance and security layer, while not having to build a Platform in house. Until now, no other open source solution had given us this type of flexibility. We're still working toward a GA release, but the speed at which we've been able to move has been incredible.
Now that there is a usable solution for complex orchestration, many newcomers will consider it the goto solution, regardless of scale. You need nginx and some python scripts? Kubernetes! You have 100 daily visitors? Kubernetes!
It’s not really their fault, it’s just a sad consequence of a convenient cloud solution.
I found that only bitbucket deployments is a good one in terms of simplicity of managing deployments (just like Heroku). Most vendors forces me to use their crappy CI solution for doing CD. Why they want me to migrate to your (very limited) build system? I can pay you for a fancy dashboard, but please, allow me to use whatever i need to build my software. There are many options on the market for CI and you just can't build them for everybody. But good CD is a way to go and easier to manage. Just give me a hook for registering for a new build (say docker image versions) and help me manage this stuff. I have tried Spinnakle, but it is too fragile for me - there are no simple way to install it (k8s even easier!) and UI is too bad for a small project.
Good CD is still missing in k8s ecosystem.
One great k8s tool I like is kompose. It allows our devs a very similar interface around secrets, networks, volumes, etc.
It's actually the same root word as cybernetics. Meaning governance. Hence tools named like Helm etc.
I believe it's the root of words like 'captain' too. Maybe even related to Caesar / Kaiser via Latin.
https://www.infoq.com/news/2015/05/mesos-powers-apple-siri
I would be surprised if they haven't done a k8s PoC at the very least though.
>Big extra points if you have experience with orchestration technologies like: - Mesos/Mesosphere - Kubernetes - Docker [Swarm]"
On the minus side, the development has been slow, with major breaking changes happening along the way. And in the latest version, the open-source version has been a bit hobbled. In particular, it doesn't support global secrets, so every project to be built has to be created and managed separately. This is not just if it needs build secrets (such as to access private Go packages, NPM modules, Ruby gems, etc.), but it's obviously also required to authenticate with a container registry. We decided to skip that since it means duplicating the same secrets for every single application. We actually reached out to the Drone guys to ask about enterprise pricing, but they didn't respond (!).
(At the moment we're back to building with Semaphore, which is a hosted solution similar to Travis. It's an old-hat CI system that spins up VMs, so it's slow and awkward to work with when it comes to Docker. But it's working okay at the moment.)
I've heard good things about GoCD [2], though. It's next on my list to investigate.
[1] https://drone.io
We have our own monitoring stack, though, so don't use the additional prometheus integrations
Great container-native pipelines and has good integration with Kubernetes, along with GKE specific hooks and the ability to run any kubectl command.
A vanishingly small number of use cases require 100's of nodes, and the accompanying orchestration. Stack Overflow runs a top 25 website on 9 webservers, and most of those are just redundancy. Nothing Ansible/Bash or Powershell/MSDeploy can't solve in orders of magnitude less time. And debugging/diagnosing problems is much easier.
Trying to integrate a k8s cluster into legacy infrastructure is non-trivial, and requires knowledgeable operations teams to deploy. Troubleshooting k8s requires extensive domain knowledge, in addition to foundational understanding of fairly advanced networking and distributed systems concepts. In my experience, working with k8s also requires much more vigilant watching of the project space for (breaking) changes, updated docs, SIG news, and github issues. Even if Google or Amazon is managing the k8s management, troubleshooting your app using kubectl, etc. is a new skill to learn.
I actually like a lot about using k8s, but I understand when people complain about the complexity.
K8S forces people to use an algebra, the expression of that algebra can take many forms. Containers are _just_ trees of processes with ... contained separation boundaries.
It's been great for us since we were productive from day one without having to worry about setting up Kubernetes or figuring out things like deployments and builds. Their documentation is great and the Ansible installer is very helpful. They have a lot of operational docs, which are invaluable (day two ops guide, among others).
Apart from the obvious PaaS features, it has a lot of small and seemingly minor improvements like "oc rsync --watch" (copy changed files to a container) or "oc rsh dc/foobar", proper auditing, and a lot of other small stuff that makes life easier.
Red Hat is one the main k8s contributors and is responsible for the RBAC implementation, among others.
You did not mention any particular pain points, but I'd love to hear about them.
Kops == AWS install Rancher == Easy small cluster on-prem or private cloud. GUI push-button setup kubeadm == CLI tool to set up a cluster manually with basic settings. Not very flexible, but can get a cluster working in minutes. kubespray == Ansible playbooks for setting up k8s cluster. The most powerful of the bunch, especially if you're familiar with ansible. This is the preferred way to run it in production I believe, along with customizations on top.
> can you point to alternatives which are simpler
Sure, there are plenty of simpler alternatives... as long as you're willing to consider ones that don't do quite as much. (Beware, you'll also need more of them.)
The complaints that Kubernetes is "complex" are obviously comparing it to more targeted solutions and ignoring the aggregate system that arises in its place.
If you find a tool with a better fit to the problem, then yay for you do the smart and simple thing. In practice, though, for many shops there's a major hodge-podge of almost-there solutions which require much more specialization on much less popular frameworks with much less vendor support to compete with Kubernetes.
Kubernetes did not come into a vacuum: it's disrupted some mature solution ecosystems by being much, much, better.
Look: they're Red Hat, I mean c'mon now. If you don't think they have great engineers then I'd hate to hear what you think of the rest of us.
These websites are not usually intended for you or I. They're intended to get the interest of the cheque-signers. Enterprise software is a very particular world and it has its particular cadences and norms and approaches. Red Hat understands it well. So does Pivotal. That's why we see so much of each other at a number of places where folks are looking to adopt modern platforms.
I feel like this one is really on you, not Red Hat.
The root issue is Centurylink has no peering with Hurricane Electric on IPv4, and ditto for peering with numerous other providers, versus on IPv6 they have a much more open peering policy. This seems to be common among a number of ISPs, though as time goes on this issue will hopefully ease.
In IPv6 land, that can introduce a lot of pointless complexity.
It would be reasonable for customers to be able to expect the ability to get /48s or /56s. Instead they don't even get a proper /64.
"You just take your app and put it in a container, then put that container in a pod, which is in a replica set with some other apps in their own containers, which is on a node in a cluster in a datacenter somewhere! Oh and the replica set is amongst a deployment. And if you want to talk to the outside world you'll need a sidecar with ngnix which is an app in a container in your pod. Or your deployment. But it's different than your app. Time to get kubectl and minikube installed..." People act like all these concepts are taught in beginner CS classes.
Kubernetes has some great attributes, but being simple isn't one. I feel sometimes that Google really pulled one on us, and this is just training people to treat the entire system as a black box, locking them in to services only top tier cloud companies have the expertise to provide and maintain reliably.
Only a few people need to know all the details. We have many developers that know absolutely nothing how this stuff works. They only know that they push code, and a container falls off the other side, which can even be installed automatically.
Also, you don't need sidecars. Add a k8s ingress, or even deploy a separate nginx that will do this for you (we use the second option). Developers do not care.
I'm starting to think that minikube is an anti-pattern.
I think you’re overestimating complexity based on lack of familiarity.
You do not have to use CoreOS, you do not need PXE boot. That's one of the possible ways.
What you do need is a handful of containers, and an overlay network. Which, if you do not want to manage, you can sign up for something like Google Cloud and let them manage it for you.
Currently running a application on anything is always complex in any kind of sense.
I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S.
I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster.
My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore.
I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...)
The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.
K8s needs a role with the correct IAM permissions to create the resources on the AWS side, it can't do it magically. Which means that someone granted these permissions before.
If anything, your InfoSec department should love that. IAM is pretty fine-grained, so they can control what gets created. Types of resources, and whatnot. Just don't grant star permissions to all of EC2.
This is approaching Terraform levels of complexity though. Unless you mean that cloud providers should do that when you create a volume, in which case I understand where you are coming from and well, they could do that. Although I'd much have them exposing a native K8s interface to take care of this PV and PVC for me and not showing me the underlying volumes at all.
Why not use Kubernetes to declaratively define nodes, disks, load balancers, firewall rules, DNS rules...? If you look at Terraform, for example, it's just an "object store" (the state file) that you modify in order to "converge" your infrastructure.
If you're already using Kubernetes via GKE or a similar IaaS, then of course Kubernetes has underlying primitives (GCE disks, GLBCs, etc.) that are exposed. But those primitives are exposed because GKE runs on top of GCP.
What I'm proposing is a "KaaS" where Kubernetes is the control plane, without the baggage of an additional API.
You don't specify a node but the CPU/RAM your application requires and let K8S scale up the cluster if necessary. Likewise, you specifiy the storage needed instead of managing disks, and set network policies to control load balancing and routing. If you want to live in Kubernetes only today, you can absolutely do that. Launch your cluster once and then use kubectl and yaml for the rest.
- I didn't even know it was called "OpenShift Origin"!
- I don't understand what the differentiation is between the various things called "OpenShift", from having visited the Google links briefly.
- Third Google hit (https://www.redhat.com/en/technologies/cloud-computing/opens...) does not even mention Origin anywhere, but describes it as "a container application platform that brings Docker and Kubernetes to the enterprise". With that copy, you already lost me. That IBM-type business site that makes my skin crawl.
- The Github site actually introduces it as "Enterprise Kubernetes for Developers", which... I don't even know what that means! What does "Enterprise Kubernetes" even mean? The actual readme's introduction ("a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment") does make sense, and the readme itself is fine. However, still no explanation of what "Origin" is compared to other things named "OpenShift".
- Openshift.org (5th link on Google) is good. That's exactly the kind of site I'd want to see as a developer/CTO. However, still no explanation of "Origin". From Openshift.org it seems like Origin is the only product.
- I go back to openshift.com and click on "Products" out of curiosity, and it's clear that there's a bunch of other OpenShifts. What are they? Why should I care? The site isn't inviting me to find out, so I leave.
Why are there are so many sites? Github is one thing, but so far we have three, and none of them really give a cohesive story.
I understand that the marketing department is targeting different people, but do they realize that people google this stuff, and will likely end up on the wrong site, with no signposts to the right thing? The fuzzy messaging leaves the impression that the company is messy and disorganized, and also pretty lost in a fog of enterprise buzzwords.
Anyway, in my original comment, what I was seeking was Kubernetes a cloud service. After sniffing around on openshift.com, I suspect the product I want is called OpenShift Online, but I'm not actually sure. Because there's also something called OpenShift Container Platform, which is also Kubernetes, but apparently in the cloud of your choice? Unfortunately, the site doesn't explain what the relationship is between this and Origin, and of course there is no pricing for Container Platform, probably because it has the label "ENTERPRISE" next to it. Figures.
There is, though; "Origin is the upstream community that powers OpenShift". Meaning Origin is the open source project upon which commercial products are built. Much like Moby is the open source project upon which Docker is built, for example.
We have a similar mix of CxO-focused and engineer-focused literature. So I guess I felt you were giving us a hard time too.
Edit: and I guess it figures that, since I work for a competing company, I'm primed with some background details on OpenShift.
Fact is InfoSec is not at liberty to set priorities for new technology development. I have to either drag the whole group with me at an institutional pace, or get someone at a higher pay grade "who will stake their career on it," in order to help me meet my goals, which really may or may not be achievable simply by means of adopting this advancement. On the other hand, lots of people who are not InfoSec like to "help" with the ostensible jobs of InfoSec, ...
which is often seen as simply delaying progress until the problem can be fully understood and the solution agreed to by a quorum? Man I can't wait that long
I'm a bit salty can you tell
Kubernetes isn't simple. Some of that is due to the problem its trying to solve, some of it is because the design, some if it is because the ecosystem that has evolved around it with regard to meshes/load balancers/control planes and overlay networks. We are after all replying on an article about a startup ecosystem being created around Kubernetes.
Tools like Kubernetes gaining adoption are partly based on technical merit/need and part based on fashion. Developers like toys and for some people the toys are what gets them through the day. Most of the people adopting Kubernetes probably don't need it. Developer Advocates and consulting shops are the ones making people think they need Kubernetes when they don't. This is an industry problem and that's all well and good. Let's just not pretend there isn't some influence from startups (typically backed by VC, which exacerbates the pressure) and consulting shops.
Sharepoint and BizTalk spring to mind... MS Dynamics (or whatever), too.
SAP and other "configuration heavy" systems bring it to another level.
> Most of the people adopting Kubernetes probably don't need it
If they geniunely don't need it then it will bite the in the ass.
On the other side: the absolute nightmare hodge-podge of half-baked non-integrated solutions that Kubernetes has murdered, and the promises it delivers on for cloud-native and cloud-first systems, are major strategic wins with clear ROI...
MS does not profile Google tech without cause. The largest virtualization supplier does not integrate competing Google tech without cause. Amazon is an easier mark here, but: the very fact there is a common, turn-key, orchestration platform across all major cloud providers is massive, and very hard to attribute to developer advocacy.
Way to trigger my PTSD...
It's a quick read, but to summarise: it's almost as complex as google's internal borg system but the benefit isn't even close (partially because nobody else has google problems). I can't down K8s personally, as I've never used it myself. But I wonder if there's the possibility of a system that's more 80/20 of google's borg rather than the seeming 20/80 coming from k8s. And I wonder if Grab will release it next year.
Borg is an accumulation of a decade's work with containers at Google, and has been described by googlers as a rich but a little messy, having been designed incrementally over many years as needs have surfaced. Borg could never be open-sourced because it's so specific to Google; for example, it uses Google's own cgroups-based container tech, not Docker/OCI/etc. Omega, as I understand, was an effort to clean up Borg and modernizing it, but apparently it was never put into production; instead, some of the innovations ended up being backported to Borg [1].
More importantly, Kubernetes is based roughly on the same design as Borg: A declarative, consistent object store, with controllers, schedulers and other bits and pieces orchestrating changes to the store, mediated by a node-local controller (Borglet/Kubelet). A major difference between Borg and Kubernetes is that with Borg, the object store is exposed to clients, whereas Kubernetes hides it behind an API. Another difference is the structure of containers; Borg's "allocs" are coarser-grained than pods and Borg is less strict about where things go, which googlers have described as a shortcoming compared to Kubernetes' strict pod/container structure. Another difference, also seen as a shortcoming, is that Borg lacks Kubernetes' one-IP-per-pod system; all apps on Borg apparently share the host's network interface. Kubernetes also innovates on Borg in several ways; for example, Borg doesn't have labels [2].
Borg, from what I gather, scales much further than Kubernetes at this point, but it's really not related to the design. The design is fundamentally the same.
Yegge's criticisms are too handwavy ("overcomplicated") to counter, but I don't think Yegge knows what he's talking about here. As for "benefit": Not sure what you mean by this, but Kubernetes arguably comes with benefits — declarative ops, platform abstractions, container isolation — even if you're just running a single node. The notion that you only need Kubernetes if you have "Google-scale problems" is just nonsense.
PS. What's "Grab"?
[1] https://ai.google/research/pubs/pub44843 (I recommend reading this paper)
[2] https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...
Grab is the company Yegge left google for. He always complained about google's inability to platformize, so random hunch is he instills this desire into Grab? But entirely random. I also don't know how influential he was inside google vs outside.
I think turnkey kubernetes solutions like Rancher will dominate for a lot of use cases, especially for individual devs and small teams that can't have a dedicated DevOps resource to manage kubernetes.
Granted we run our own bootstrap (when we started none of the bootstraps running around we're ready, we started with a terraform/make implementation of hightower's hardway and we've just kept adding. We're thinking about revisiting the space again.
You don't go Google scale because it's cool to. You either do it because you absolutely have to, or you don't because, thankfully, you don't have to.
Attaching the Google name to projects stopped having meaning ~5 years ago. Their hiring process optimizes for fresh graduates who know nothing about engineering. Something coming out of Google means nothing, especially when Google itself doesn't even use it!
>It was created by engineers for engineers.
So was openstack. Then the feature creep happened.
I'm currently supporting ops of some "eventually consistent, globally distributed" (if you can fathom through the previous engineer's algorithms) built-from-scratch system designed to be resilient in the face of multi-megaX-transactions-per-second, that currently contains all of 8K records.
Whereas stackoverflow.com runs on SQL. Their own hardware at that.
I half wonder if a large chunk of AWS's revenue could be replaced by https://www.amazon.co.uk/electrical-sockets-adhesive-sticker...
Pick any large tech company, they have thousands of servers and countless customized software running. That's what is meant by google scale.
They need to manage that and they suck at it most of the time, they don't know what resources they own and they can't figure out what's running or where.