Etcd is what holds k8s together. I saw the concept of watchers evolve from what-used-to-considered a fringe use case to something so fundamental and powerful in k8s.

The only issue is that its usage best practices are not followed. Most users don’t back up etcd, using it like persistent storage. Resultantly, in a few upgrade cycles, nobody really knows what’s store in the etcd store and it becomes a “super pet”.

This article does a good job laying out the best practices. I hope users start adopting these too.