Framework for 0 downtime deploys
Confidential
Hello All
I wonder if I could garner some of the wealth of experience on this group:
Our current application deployments (every 3 weeks) require about 30min downtime. We are now tasked of making this 0 downtime.
From all the reading I have done, we have solutions for the infrastructure and code deploy, but with regards to the DB the main issue seems to be keeping the new deploy backwards compatible - functions/tables/fields - all of it.
That seems like quite a large management task and would require careful reviews of changes. Is there any type of framework that already manages this type of capability? Or are there aspects of PG that we should be using in this regard?
Thx
Z
No silver bullets exist (yet). A couple of things I can recommend:
1. GitLab's experience of changing schema without downtime and maintaining
backward compatibility – they have open documentation and a lot of things
solved and documented
- start here:
https://docs.gitlab.com/ee/development/migration_style_guide.html
- check their migration_helpers.rb, plenty of useful things there
2. What my team and I are doing with respect to database migration testing:
https://postgres.ai/. We created Database Lab Engine, an open-source tool
to clone databases of any size in seconds and test, manually or
automatically in CI, anything you want using "full-size" databases. It can
help you catch and block dangerous changes leading to downtime, as well as
(if you have a well-maintained set of tests for CI) enforce the backward
compability.
Nik
On Sat, May 22, 2021 at 2:12 PM Zahir Lalani <ZahirLalani@oliver.agency>
wrote:
Show quoted text
Confidential
Hello All
I wonder if I could garner some of the wealth of experience on this group:
Our current application deployments (every 3 weeks) require about 30min
downtime. We are now tasked of making this 0 downtime.From all the reading I have done, we have solutions for the infrastructure
and code deploy, but with regards to the DB the main issue seems to be
keeping the new deploy backwards compatible – functions/tables/fields – all
of it.That seems like quite a large management task and would require careful
reviews of changes. Is there any type of framework that already manages
this type of capability? Or are there aspects of PG that we should be using
in this regard?Thx
Z
On Sat, May 22, 2021 at 11:11:46AM +0000, Zahir Lalani wrote:
Confidential
Hello All
I wonder if I could garner some of the wealth of experience on this group:
Our current application deployments (every 3 weeks) require about 30min
downtime. We are now tasked of making this 0 downtime.
I suggest you decide how much downtime you can accept or consider
multi-master replication:
https://momjian.us/main/blogs/pgblog/2018.html#December_24_2018
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Confidential
Thank you Nik
That’s very useful and will be checking out the migration test tool!
Z
From: Nikolay Samokhvalov <samokhvalov@gmail.com>
Sent: 22 May 2021 13:32
To: Zahir Lalani <ZahirLalani@oliver.agency>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Framework for 0 downtime deploys
No silver bullets exist (yet). A couple of things I can recommend:
1. GitLab's experience of changing schema without downtime and maintaining backward compatibility – they have open documentation and a lot of things solved and documented
- start here: https://docs.gitlab.com/ee/development/migration_style_guide.html
- check their migration_helpers.rb, plenty of useful things there
2. What my team and I are doing with respect to database migration testing: https://postgres.ai/. We created Database Lab Engine, an open-source tool to clone databases of any size in seconds and test, manually or automatically in CI, anything you want using "full-size" databases. It can help you catch and block dangerous changes leading to downtime, as well as (if you have a well-maintained set of tests for CI) enforce the backward compability.
Nik
On Sat, May 22, 2021 at 2:12 PM Zahir Lalani <ZahirLalani@oliver.agency<mailto:ZahirLalani@oliver.agency>> wrote:
Confidential
Hello All
I wonder if I could garner some of the wealth of experience on this group:
Our current application deployments (every 3 weeks) require about 30min downtime. We are now tasked of making this 0 downtime.
From all the reading I have done, we have solutions for the infrastructure and code deploy, but with regards to the DB the main issue seems to be keeping the new deploy backwards compatible – functions/tables/fields – all of it.
That seems like quite a large management task and would require careful reviews of changes. Is there any type of framework that already manages this type of capability? Or are there aspects of PG that we should be using in this regard?
Thx
Z