Achieving a zero-downtime Postgres major version upgrade

Achieving a zero-downtime Postgres major version upgrade(medplum.com)

15 points by mattlong 1 year ago | 3 comments

Hey HN,

I put together this blog post[0] and this repository[1] with supporting resources to immortalize the decision tree and step-by-step run book I used that led to us successfully upgrading the AWS RDS Aurora Postgres database of Medplum's hosted service from v12 to v16 without downtime. There are many write-ups out there on the topic, but just about all of them were frustratingly light on details or assumed the reader was already fluent in the technologies used. This guide and the runbook are anything but light on details and attempts to explain/describe decisions and steps from first principles when possible.

As just about anyone who has worked with Postgres version upgrades and logical replication will tell you, there are many sharp edges and a steep learning curve that can only be overcome by doing. I have varying levels of experience with the tools used and by no means claim to be an expert in any of them! I'm sure there are ways this process could be improved and streamlined and would absolutely love to hear any suggestions and field questions from you either here or in the issues/discussions/PRs of the repository.

Medplum[2] relies heavily on Postgres and we only intend on deepening our relationship with it and all it has to offer as we continue to grow. That was a big part of our reasoning to justify investing the time to level-up our operational proficiency with Postgres. We're an open source company, so it felt right to make this guide open source too!

[0]: https://www.medplum.com/blog/zero-downtime-postgres-major-ve... [1]: https://github.com/medplum/medplum-postgres-upgrade [2]: https://github.com/medplum/medplum

gabrielruttner 1 year ago |

Hey - great writeup!

Curious how you're planning to handle Postgres versions going forward - will v16 be a hard requirement for new OSS deployments? What's your strategy for managing existing deployments on older versions?

mattlong 1 year ago | |

Thank you! We generally try to stay in line with the supported versions of the technologies we use: Postgres, NodeJS, Redis, etc.

For Postgres in particular, part of the motivation for doing this upgrade when we did was (besides the very meaningful performance improvements) is that Postgres 12 will no longer be supported by Aurora after February 25, 2025.[0]

Officially, Medplum v3 (our current newest version) requires Postgres 12+. We're just about to release v4 which will support 13-17 based on the support dates published by Postgres[1]. We're still finalizing more formal documentation on our versioning strategy going forward.

[0]. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPostgreSQ...

[1]. https://www.postgresql.org/support/versioning/

brown 1 year ago |

Love Postgres, but major version upgrades remain the one genuine pain point. One misstep and you're in backup-restoration hell. Solid guide.