I'm lazy and I don't like having to remember "the right way" to run something, so my solution is directories and wrappers. I keep a directory for every environment (dev, stage, prod, etc) for every account I manage.

  env/
      account-a/
                dev/
                stage/
                prod/
      account-b/
                dev/
                stage/
                prod/

I keep config files in each directory. I call a wrapper script, cicd.sh, to run certain commands for me. When I want to deploy to stage in account-b, I just do:

  ~ $ cd env/account-b/stage/
  ~/env/account-b/stage $ cicd.sh deploy
  cicd.sh: Deploying to account-b/stage ...

The script runs ../../../modules/deploy/main.sh and passes in configs from the current directory ("stage") and the previous directory ("account-b"). Those configs are hard-coded with all the correct variables. It's impossible for me to deploy the wrong thing to the wrong place, as long as I'm in the right directory.

I use this model to manage everything (infrastructure, services, builds, etc). This has saved my bacon a couple times; I might have my AWS credentials set up for one account (export AWS_PROFILE=prod) but trying to deploy nonprod, and the deploy immediately fails because the configs had hard-coded values that didn't match my environment.

Agree, this is a huge pain point when dealing with multiple clusters. I wrote a wrapper for `kubectl` that displays the current context for `apply` & `delete` and prompts me to confirm the command. It's not perfect, but it's saved me a lot of trouble already — but encouraging other members of the team to have a similar setup is another story.

Here's the script (along with a bunch of extra utils): https://github.com/pch/dotfiles/blob/master/kubernetes/utils...

A lot of people like kubectx. Or specifying contexts. Personally I hate both approaches.

For the several dozen clusters that I manage, I have separate kubeconfig files for each and I use the --kubeconfig flag.

It's explicit and I have visual feedback in the command I run for the cluster I'm running against, by short name. No stupidly long contexts.

Not a perfect solution but I add a prompt signaling both my current namespace and cluster, along with some safeguards for any changes on our production environment. In practice I haven't deployed something wrongfully in production ever.

I use a custom written script but I've used this one in the past - its pretty nice.

https://github.com/jonmosco/kube-ps1/blob/master/kube-ps1.sh

We partially resolve this by having different namespaces in each of our environments. Nothing is ever run in the 'default' namespace.

So if we think we're targeting the dev cluster and run 'kubectl -n dev-namespace delete deployment service-deployment' but our current context is actually pointing to prod then we trigger an error as there is no 'dev-namespace' in prod.

Obviously we can associate specific namespaces to contexts to traverse this safety net but it can help in some situations.

direnv is our magic sauce for this. We enforce that all devs store the current context in an environment variable (KUBECTL_CONTEXT), and define the appropriate kubectl alias to always use that variable as the current context. To do stuff in a cluster, cd into that cluster’s directory, and direnv will automatically set the correct context. I also change prompt colors based on the current context.

(This way, the worst you can do is re-apply some yaml that should’ve already been applied in that cluster anyway)

We also have a Makefile in every directory, where the default pseudo-target is the thing you want 99% of the time anyway: kustomize build | kubectl apply -f -

I wrote a convoluted tool for this problem which isolates kubectl environments in docker containers: https://github.com/forestgagnon/kparanoid.

This approach allows the convenience of short, context-free commands without compromising safety, because the context info in the shell prompt can be relied on, due to the isolation.

There are some things which don't work well inside a docker container (port-forwarding for example), but it does make it simple to have isolated shell history, specific kubectl versions, etc.

When I was running the internal k8s clusters at a previous workplace, I simply got into the habit of compulsively running `kubectl config current-context` to check which one of the 50+ clusters I was currently connected to (designated test clusters for *playing with cluster infra", designated clusters for "devs playing around", designated prod clusters, with segregation between "batch-like" and "interactive" workloads, as we needed to treat the nodes differently in those, designated "run the CI/CD pipelines" clusters, as they needed different RBAC, ... and then duplicate between multiple data centres).

thanks for starting that thread, context is a major hurdle for beginners.

I myself am quite happy with the basics, but have an alias on k=kubectl and set-context that without argument displays the current-context. Before doing anything I rename or edit contexts in .kube/config to have a minimal amount of characters to type for the target ("proj-prod"). Using -l name= is another help in filtering, jsonpath and jq too.. as years ago with using the cli prompts with database products, building up muscle memory also gave me opportunity to grok the concepts at the same time.

After some attempts with different tooling, I came to like kubernetes for what it can do.

k9s is a fantastic tool. It's a CLI GUI written in go.

I like the spirit of this but for dealing with multiple clusters, kubectx is pretty standard, always returns highlighting where you are and we don't have to type in the cluster name in every command. Also avoiding "kubectl delete" seems such a narrow case, I can still delete with "k scale --replicas=0" and possibly many other ways; at this point you are better of with a real RBAC implementation.

Every member of our ops team has the following PS1 var in .bashrc to prevent such accidents:

  PS1='\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]\[\033[01;33m\] [`kubectl config current-context| rev | cut -d_ -f1 | rev`] \[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\] $ '

couldn't that be solved by not allowing production access to those clusters? most k8s providers should allow role based access (read/write/deploy)

Nice trick, I usually use sleep 10000000 as the command.

Not only containers, it can also forward services!

omg TIL `k explain foo` and `k explain foo.spec` is a thing. thank you for this!!!

Wait What

  <type>.<fieldName>[.<fieldName>]
  
 Add the --recursive flag to display all of the fields at once without descriptions. Information about each field is

  # Get the documentation of a specific field of a resource
  kubectl explain pods.spec.containers

I could be wrong here, but I think `helm lint` just checks that the chart is formed correctly — Go templating and all.

I don't think it validates the Kubernetes resources.

Here's an example:

$ helm create foo

$ cd foo

Then change "apiVersion" in deployment.yaml to "apiVersion: nonsense"

In the linting, I got

$ helm lint ==> Linting . [INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

$ helm template . | kubeval -

ERR - foo/templates/deployment.yaml: Failed initializing schema https://kubernetesjsonschema.dev/master-standalone/deploymen...: Could not read schema from HTTP, response status is 404 Not Found

Along those lines, this was an interesting statement:

"You should learn how to use these commands, but they shouldn't be a regular part of your prod workflows. That will lead to a flaky system."

It seems like there's some theory vs. practice tension here. In theory, you shouldn't need to use these commands often, but in practice, you should be able to do them quickly.

How often is it the case in reality that a team of Kubernetes superheroes, well versed in these commands, is necessary to make Continuous Integration and/or Continuous Deployment work?

Why wouldn't that be determinstic? You should be using a service for that.

    kubectl get svc -l app=<your-app-name>

For the port is trivial: `kubectl get pod <yourpod> --output jsonpath={.spec.ports[*].port}` or if you don't remember the json path just `k get pod <yourpod> |grep Port`.

For the IP address, why do you need that? with k8s dns you can easily find anything by name.

You can use fieldSelector

https://stackoverflow.com/questions/39231880/kubernetes-api-...

It seems this may have been posted on the wrong article, as it's completely unrelated to the topic.

Both AWS and GCP allow you to customise a machine's spec. You're not limited to the ones they offer.

Wow, I’d forgotten about this. The reason no one has fixed it is partially because I didn’t do a great job of describing what the fix was I expected to see (clarified now). Reopened and will poke folks to look.

> I'm pretty sure if you have the time to make a PR to fix it, it would be welcome.

Google had a net income of $17.9 billion in just Q1 of 2021.

I believe they have the resources to fix that, and I will not be shamed into "if you have time, please open a PR towards this opensource project".

> But I'm guessing it's non trivial or it would have been fixed by now - probably a quirk of the code generation logic.

I remember the "quirks of generation logic" being used as an excuse for Google's horrendous Java APIs towards their cloud services. "It's just how we generate it from specs and don't have the time to make it pretty".

For the life of me can't find that GitHub issue that called this out. Somehow their other APIs (for example, .net) are much better.

Edit: found it

https://github.com/googleapis/google-cloud-java/issues/2331#... and https://github.com/googleapis/google-cloud-java/issues/2331#...

> If what you want to deploy is best described as “an application” it’s probably not the right tool for the job. If what you want to deploy is best described as “50 interconnected applications” it’s probably going to save you time.

This is an excellent way of looking at it. I've struggled for many years to come up with a response to hacker news comments saying you don't need kubernetes, but this sums it up about as well as I could imagine.

As someone who runs both in production, Nomad would almost certainly meet your needs.

Learning how to operate Kubernetes well takes a while and I would say is only worth the investment for a extremely tiny percentage of companies.

Modern istio provides a lot of value to a single application. mTLS security, telemetry, circuit breaking, canary deployments, and better external authentication and authorization. I’ve seen each done so many different ways. Nice to do it once at the mesh layer and have it be done for everything inside the cluster.

This kind of comment is on every single HN post about Kubernetes and is tiresome. I also think it's off topic (TFA is about kubectl tricks, not about the merits of K8s).

We get paid because we know these tools. It's why we're desired: because the company thinks they want K8s or they're one foot in EKS and they're doubling down. We don't get hired because we dare to suggest they dismantle their pilot cluster and take a sharp turn into Nomad.

Most of us aren't the engineering heads of our departments. So you'll forgive us if we continue pushing the moneymakers we have in our heads and setting up our homelab clusters. I want to be paid, I want to be paid well. It may as well be pushing the technology stack that scales to megacorps because who knows maybe I'll make it there one day.