How to take down production with a single Helm command

How to take down production with a single Helm command(ounapuu.ee)

44 points by hddherman 2 years ago | 16 comments

psini 2 years ago |

Tangentially related, cannot help but be reminded of the "Ohnosecond": https://en.wikipedia.org/wiki/List_of_humorous_units_of_meas...

silisili 2 years ago |

I've seen enough of these over the last two decades, between Ubuntu/Debian and RedHat, that I use and inherently trust other distros more. Just compile the source packages. If there is some issue, file a bug. Monkey patching everything silently leads to things like this, which I've never experienced with distros like Arch, Void, or Solus.

sneak 2 years ago | |

Debian is very very diligent about patching out phone-home and expiration timers and other such “developer knows best” misfeatures. Some apps also include autoupdate (aka solarwinds style RCE) which they also remove.

Overall they do way more good than harm. Trashing them because they sometimes make mistakes is probably incorrect.

p_l 2 years ago | | |

But when they get it wrong, they can get it catastrophically wrong - like breaking ssh-keygen so that for given set of parameters there existed only 2^32 keys.

vbezhenar 2 years ago | |

For helm and most other golang apps, you just need to download and put binary into your $PATH. They usually put them onto github releases, so it's really low friction way to install a genuine application.

Now to keep them updated is another story.

silisili 2 years ago | | |

Last sentence. I love Go's install and run story, but keeping it up to date is a pain.

We trust, or used to trust, distros for this. Why are they messing with the source?

ofrzeta 2 years ago |

That's a good argument for not running IaC tools on your own workstation but on a dedicated machine where the environment is also somehow vetted.

On the other hand I am not sure how to actually vet all used tools. For instance if you take Helm there are many pointers to repos for various Linux distros and other OSs: https://helm.sh/docs/intro/install/

Who to trust? While the Helm page says "members of the community" are in charge of the Ubuntu repo the Fedora repo is called an "official repository". Presumably judging from the OP that means "official Fedora repo" opposed to "official Helm repo".

So essentially there are no repos of the Helm maintainers which makes installing and auto-updating it on any Linux distro virtually impossible.

adrianmsmith 2 years ago |

Explanation here: https://github.com/helm/helm/issues/12681#issuecomment-19593...

Looks like it's a bug in Helm, but actually isn't Helm's fault, the issue was introduced by Fedora Linux.

(This issue is linked from the article, but it took me some time to find it.)

olabyne 2 years ago | |

why would someone use Fedora in production, tho ?

vbezhenar 2 years ago | | |

helm runs on the operator machine, not server. While fedora server might not be very popular, fedora workstation seems to be pretty popular.

rad_gruchalski 2 years ago |

> Apparently the version of Helm packaged in Fedora Linux included a patch that introduced this issue

Let's translate: how to take down whatever by living on the bleeding edge.

readthenotes1 2 years ago | |

And increase the bloodshed by, apparently, configuring your staging environment significantly different than your production environment. An instigating the bloodshed by doing something first in production and then testing it in staging (sure, the first time staging wouldn't have caught it, but maybe then you could look to see where you misconfigured staging...)

number6 2 years ago | |

Fedora is hardly bleeding edge

zer00eyz 2 years ago | | |

I always thought of Fedora as QA for red hat.

rad_gruchalski 2 years ago | | |

Apparently bleeding enough in this case.