Ask HN: How to do rolling deployments without Kubernetes? Every month, there seems to be a new post stating ~"don't use Kubernetes yet", but what should we use instead if we cannot have downtime? |
Ask HN: How to do rolling deployments without Kubernetes? Every month, there seems to be a new post stating ~"don't use Kubernetes yet", but what should we use instead if we cannot have downtime? |
The requirements:
- Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar
- You need to have something in front of your application, like nginx or apache
- Whatever webserver you have should be able to "hang"/pause/suspend requests while you switch application version
- Each version you create needs to be concerned about what the previous version did. Breaking changes needs to happen across multiple versions, where you can soft-deprecate something, and a release after that, actually "break" it
- You need to have some sort of "healthchecking" that can tell you if the new version is OK or not
The implementation:
- You have Version 1 running of your backend on port X
- You want to deploy the new version, so you deploy it to your server but the web server in front still serves requests from Version 1
- Run healthchecks against Version 2
- Once they pass, tell web server to "pause" in-flight requests
- Switch web server configuration to use Version 2 application instead (this can be combined with the previous step, `nginx reload` would combine these for example)
- Stop Version 1 from running
- Repeat for each new version
And now you've achieved deploying a new version of your application without any failing requests.
(Sidenote: You can replace hanging/pausing requests with graceful shutdown of your webserver (meaning it waits for no pending requests) if you have a low amount of traffic)
I wish more people approached their software engineering teaching in this way.
Thank you for the great comment.
Me too, so why not do it myself? :) Thanks for the feedback
Anyway, how we’ve done it is by setting up a pipeline, that builds and deploys a docker container of a repository to a deployment a lot on a cloud application/function/whatever you call them if it passes the build phase. Then depending on whether or not the deployment passes the criteria set up for it, it either automatically swaps deployment slots with the production slot or waits for manual confirmation to do so. On anything that is allowed a reload, we don’t swap deployment slots but instead redeploy, typically with minimal downtime.
The reason it’s annoying is because it takes a lot of time to set up for each pipeline, and when you need to be able to swap deployment slots with now downtime you also need to handle things like build processes taking global variables or clients needing to be told to reload parts or them.
I’m not sure kubernetes is the answer for us, but we’re certainly going to look for a way to make the whole process smarter as it sometimes takes up more time to setup the pipeline and deployment environment than building the service that needs to be deployed.
Controlling the upstream traffic via haproxy/nginx allowed Ops to roll out blue-green, canary and waterfall/rolling deployment methods albeit with more human interaction.
Digital Ocean have this [0] article from 2014 on load balancing.
[0] https://www.digitalocean.com/community/tutorials/an-introduc...
We did this at work and works fine (blogpost will be out soon). It allows us to decommission a cluster by removing the cluster from the LB. There's nothing new about this technique.
We're using DOKS + CloudFlare Traffic but you can use any LB service (no affiliation, as all products there are pro's and con's).
Once the setup is ready, operations are easy:
a) Remove cluster from the LB
b) Perform cluster operations (ingress upgrade, k8s upgrade, possibly disruptive daemonset operations)
c) Add cluster to LB
Another pro is that when a region has cloud-provider level issues (happened with FRA1 few days ago) we can remove the cluster from the LB and stop worrying about it until the the issue is fixed. LBs have health-checks and such, to automate addition/remove of clusters.
You either deploy from command line or something like a GitHub action (or CI).
Container 1 and 2 are briefly running concurrently, and networking is updated to point to the new container once it finishes its build phase, and passes any pre-defined tests (or after 10 seconds of running without crashing).
a) remove server(s) from load balancing, finish requests/connections in progress, restart with new software, add to load balancing
b) hot loading
c) start a new server instance and pass it the listen socket (or if your OS isn't great, drop syns for a bit while you close the listen socket on the old server and open a new listen socket for the new server)
I like hotloading, but it's not appropriate for all updates, so you need to have a way to handle restart based updates as well.
I.e. github action to do your build, test. Dont do the docker push to the repository unless it passes the tests. Always do your docker push from the correct branch. If you need approval for release, build that into the pipeline.
Azure pipelines pretty much the same deal.
You don't need kubernetes for a few services, it only gets useful when you have a lot.
https://cloud.google.com/blog/products/serverless/cloud-run-...
I think the idea is worth mentioning because people often think it's hard to achieve / something special. But there's a hundred ways to do seamless version swap on a host.