Remove Deprecated Exclusive Backup Mode

On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:

Non-exclusive backups have been available since 9.6 and several third-party
solutions support this mode, in addition to pg_basebackup.

I think that two releases is not actually that much time to just nuke
the exclusive backup interface. I would be fine if the docs show the
deprecation more aggressively using a warning section, and we could add
an explicit WARNING message about the matter directly when calling
pg_start_backup and pg_stop_backup.

My 2c.
--
Michael

Hi,

On 2018-11-27 12:20:13 +0900, Michael Paquier wrote:

On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:

Non-exclusive backups have been available since 9.6 and several third-party
solutions support this mode, in addition to pg_basebackup.

I think that two releases is not actually that much time to just nuke
the exclusive backup interface. I would be fine if the docs show the
deprecation more aggressively using a warning section, and we could add
an explicit WARNING message about the matter directly when calling
pg_start_backup and pg_stop_backup.

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

Greetings,

Andres Freund

On Mon, Nov 26, 2018 at 10:13 PM David Steele <david@pgmasters.net> wrote:

I propose we remove the deprecated exclusive backup mode of
pg_start_backup()/pg_stop_backup() for Postgres 12.

-1. I don't have a problem with deprecating exclusive backup mode
eventually, but I don't see any good reason to do it this soon.

It's not like the problems with exclusive backup are so serious that
you can't work around them. If you know which machine is your master,
you can arrange to remove backup_label on reboot (only) on the master
(only). Sure, a lot of people probably do this wrong, but it's
infeasible to disallow all the things that people might use
incorrectly without making the system useless.

There must be hundreds or thousands of backup scripts written by
individual users that still use exclusive mode floating around out
there. Forcing all of those to be updated or scrapped will annoy
users to no benefit.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Nov 27, 2018 at 4:46 AM Andres Freund <andres@anarazel.de> wrote:

Hi,

On 2018-11-27 12:20:13 +0900, Michael Paquier wrote:

On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:

Non-exclusive backups have been available since 9.6 and several

third-party

solutions support this mode, in addition to pg_basebackup.

I think that two releases is not actually that much time to just nuke
the exclusive backup interface. I would be fine if the docs show the
deprecation more aggressively using a warning section, and we could add
an explicit WARNING message about the matter directly when calling
pg_start_backup and pg_stop_backup.

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

Yeah, I'm in the same boat here -- it feels like it's a bit too short, but
since we're breaking things for people *anyway*, it's probably better to
break both at once than to have all those people have their things broken
multiple times.

--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/&gt;
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/&gt;

On 11/26/18 11:04 PM, Robert Haas wrote:

On Mon, Nov 26, 2018 at 10:13 PM David Steele <david@pgmasters.net> wrote:

I propose we remove the deprecated exclusive backup mode of
pg_start_backup()/pg_stop_backup() for Postgres 12.

-1. I don't have a problem with deprecating exclusive backup mode
eventually, but I don't see any good reason to do it this soon.

It's not like the problems with exclusive backup are so serious that
you can't work around them. If you know which machine is your master,
you can arrange to remove backup_label on reboot (only) on the master
(only).

I do not recommend that users write automated scripts to delete files
out of $PGDATA. Accidents happen that way.

If this is our recommendation on how the mitigate the problem with
exclusive backup then I think it proves the point that it is broken and
should be removed.

Sure, a lot of people probably do this wrong, but it's
infeasible to disallow all the things that people might use
incorrectly without making the system useless.

But we have already fixed this problem -- it only remains to remove the
old method.

There must be hundreds or thousands of backup scripts written by
individual users that still use exclusive mode floating around out
there. Forcing all of those to be updated or scrapped will annoy
users to no benefit.

But we are OK with forcing them to scrap their recovery scripts?

And there is a benefit -- Postgres will be able to restart if it crashes
during a backup.

Regards,
--
-David
david@pgmasters.net

On 27/11/2018 04:46, Andres Freund wrote:

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

But backup scripts are not affected by the recovery.conf changes.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Nov 27, 2018 at 02:21:49PM +0100, Peter Eisentraut wrote:

On 27/11/2018 04:46, Andres Freund wrote:

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either. Or perhaps pgBackRest has a dependency in this area?
--
Michael

On Tue, 27 Nov 2018 at 03:13, David Steele <david@pgmasters.net> wrote:

The deprecated exclusive mode promises to make a difficult problem
simple but doesn't live up to that promise. That's why it was replaced
externally in 9.6 and why pg_basebackup has not used exclusive backups
since it was introduced in 9.2.

Non-exclusive backups have been available since 9.6 and several
third-party solutions support this mode, in addition to pg_basebackup.

The recently introduced recovery changes will break current automated
solutions so this seems like a good opportunity to make improvements on
the backup side as well.

I'll submit a patch for the 2019-01 commitfest.

-1 for removal, in this release

It's not there because anyone likes it, it's there because removing it is a
risk

Recent changes are the reason to keep it, not a reason to remove.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/&gt;
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Michael Paquier (michael@paquier.xyz) wrote:

On Tue, Nov 27, 2018 at 02:21:49PM +0100, Peter Eisentraut wrote:

On 27/11/2018 04:46, Andres Freund wrote:

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either. Or perhaps pgBackRest has a dependency in this area?

If you don't consider your recovery scripts and your backup scripts to
be related then I've really got to wonder how you're regularly testing
your backups to make sure that they're actually valid.

If you aren't regularly testing your backups then I've got little
sympathy.

To be clear, pgbackrest doesn't have any dependency here- but it, like
all of the other 3rd party backup solutions and any restore solution
that a user has come up with, are going to have to be changed to deal
with the changes in how recovery works, so this is a good time to make
these changes.

Thanks!

Stephen

Greetings,

* Robert Haas (robertmhaas@gmail.com) wrote:

There must be hundreds or thousands of backup scripts written by
individual users that still use exclusive mode floating around out
there. Forcing all of those to be updated or scrapped will annoy
users to no benefit.

How about automated recovery scripts?

We've decided it's fine to break all of those which exist out there.

I'm concerned, seriously, that people don't have anywhere near the
concern about the recovery side of things as they do about the backup
side of things and that's really concerning.

Thanks!

Stephen

On 27/11/2018 15:45, Stephen Frost wrote:

But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either. Or perhaps pgBackRest has a dependency in this area?

If you don't consider your recovery scripts and your backup scripts to
be related then I've really got to wonder how you're regularly testing
your backups to make sure that they're actually valid.

The sort of installations that continue to use the exclusive backup mode
probably have the following tooling: a 20-line shell script to make the
backup and either a 10-line shell script or a similarly sized README or
wiki page to do the recovery. Changing the latter for the recovery.conf
changes is probably a 3-line change. Changing the former for the
removal of exclusive backups would require major changes. (Try writing
a shell script that keeps a psql session open while it takes the backup
from the file system. It's possible, but it requires significantly more
logic.)

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:

On 27/11/2018 15:45, Stephen Frost wrote:

But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either. Or perhaps pgBackRest has a dependency in this area?

If you don't consider your recovery scripts and your backup scripts to
be related then I've really got to wonder how you're regularly testing
your backups to make sure that they're actually valid.

The sort of installations that continue to use the exclusive backup mode
probably have the following tooling: a 20-line shell script to make the
backup and either a 10-line shell script or a similarly sized README or
wiki page to do the recovery. Changing the latter for the recovery.conf
changes is probably a 3-line change. Changing the former for the
removal of exclusive backups would require major changes. (Try writing
a shell script that keeps a psql session open while it takes the backup
from the file system. It's possible, but it requires significantly more
logic.)

They're also the sort of installations which don't have reliable backups
and don't have any clue about the danger they are in due to the current
bug/issue/whatever we have with exclusive backups.

No, I don't agree that it's sensible to continue to march on as if
nothing is wrong.

Thanks!

Stephen

On 11/27/18 3:46 PM, Stephen Frost wrote:

I'm concerned, seriously, that people don't have anywhere near the
concern about the recovery side of things as they do about the backup
side of things and that's really concerning.

I agree with your larger point, but in this case the two breakages do
not seem equal. As far as I gather the removal of recovery.conf will in
practice result in a longer downtime when your recovery scripts breaks
but not any data loss. While if we remove the exclusive backup mode then
some people's backup scripts will break and if they do not properly
monitor their backups they risk data loss.

I think we should use more caution when data loss is at stake rather
than "just" downtime. So personally I am in favor of updating the manual
with warnings (right now it does not even say if exclusive or
non-exclusive is the default) and adding a deprecation warning when
people use the exclusive mode.

Best regards,
Andreas

Greetings,

* Andreas Karlsson (andreas@proxel.se) wrote:

On 11/27/18 3:46 PM, Stephen Frost wrote:

I'm concerned, seriously, that people don't have anywhere near the
concern about the recovery side of things as they do about the backup
side of things and that's really concerning.

I agree with your larger point, but in this case the two breakages do not
seem equal. As far as I gather the removal of recovery.conf will in practice
result in a longer downtime when your recovery scripts breaks but not any
data loss. While if we remove the exclusive backup mode then some people's
backup scripts will break and if they do not properly monitor their backups
they risk data loss.

Let's please try to remember that this is across a major version upgrade
and is removing a broken capability that's already been deprecated for a
couple years.

If they aren't monitoring their backup scripts today, and aren't
regularly performing restores of those backups, they're already risking
data loss.

I think we should use more caution when data loss is at stake rather than
"just" downtime. So personally I am in favor of updating the manual with
warnings (right now it does not even say if exclusive or non-exclusive is
the default) and adding a deprecation warning when people use the exclusive
mode.

Waiting isn't going to change any of these factors, nor will throwing
warnings about the exclusive mode if people aren't monitoring their
scripts.

If we really want to keep the exclusive backup mode around, then, as
Magnus said on a nearby thread, it needs to be fixed. If it's fixed and
just works and everyone's scripts work, then great, then we can
un-deprecate it and move on.

If we aren't going to fix it then let's remove it.

Thanks!

Stephen

On 11/27/18 4:11 PM, Stephen Frost wrote:

I agree with your larger point, but in this case the two breakages do not
seem equal. As far as I gather the removal of recovery.conf will in practice
result in a longer downtime when your recovery scripts breaks but not any
data loss. While if we remove the exclusive backup mode then some people's
backup scripts will break and if they do not properly monitor their backups
they risk data loss.

Let's please try to remember that this is across a major version upgrade
and is removing a broken capability that's already been deprecated for a
couple years.

I know that and you know that, but even our own manual use the exclusive
mode in some of the examples, e.g: "pg_start_backup('label_goes_here')"[1].

Your point about that people who do not monitor their backups wont
notice warnings is fair, but even so I think we should give people more
time to migrate when even our own documentation isn't always clear about
the exclusive mode being deprecated.

1.
https://www.postgresql.org/docs/11/functions-admin.html#FUNCTIONS-ADMIN-BACKUP

Andreas

On 27/11/2018 16:02, Stephen Frost wrote:

They're also the sort of installations which don't have reliable backups
and don't have any clue about the danger they are in due to the current
bug/issue/whatever we have with exclusive backups.

No, I don't agree that it's sensible to continue to march on as if
nothing is wrong.

It's fair to argue that this facility is broken and needs to be removed.
But that needs to be its own argument.

The argument in this subthread is that since we're already making
changes in an adjacent area, removing this feature will have a low impact.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 11/27/18 9:54 AM, Peter Eisentraut wrote:

On 27/11/2018 15:45, Stephen Frost wrote:

But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either. Or perhaps pgBackRest has a dependency in this area?

If you don't consider your recovery scripts and your backup scripts to
be related then I've really got to wonder how you're regularly testing
your backups to make sure that they're actually valid.

The sort of installations that continue to use the exclusive backup mode
probably have the following tooling: a 20-line shell script to make the
backup and either a 10-line shell script or a similarly sized README or
wiki page to do the recovery. Changing the latter for the recovery.conf
changes is probably a 3-line change.

Really? We have gone from a file that can be safely overwritten
(recovery.conf) to a file that might need to be parsed to figure out if
the settings already exists (postgresql.auto.conf). Of course, you can
just continue to append to the file but that seems pretty grotty to me.

This will also require changes to all HA solutions or backup solutions
that deal with recovery. I think you underestimate how big a change
this is. I've been thinking about how to write code for it and it is
significantly more complex -- also more flexible so I'm happy about that.

Changing the former for the
removal of exclusive backups would require major changes. (Try writing
a shell script that keeps a psql session open while it takes the backup
from the file system. It's possible, but it requires significantly more
logic.)

We provide pg_basebackup with core and it is a solid tool for doing
backups. Do we really want to encourage the use of bash scripts to do
what we already have a tool for? If users want to do something more
complex than pg_basebackup then bash is certainly not the tool for that.

Regards,
--
-David
david@pgmasters.net

On 11/27/18 10:29 AM, Peter Eisentraut wrote:

On 27/11/2018 16:02, Stephen Frost wrote:

They're also the sort of installations which don't have reliable backups
and don't have any clue about the danger they are in due to the current
bug/issue/whatever we have with exclusive backups.

No, I don't agree that it's sensible to continue to march on as if
nothing is wrong.

It's fair to argue that this facility is broken and needs to be removed.
But that needs to be its own argument.

I believe I made this argument in the OP.

The argument in this subthread is that since we're already making
changes in an adjacent area, removing this feature will have a low impact.

Fair enough, but I think the argument against exclusive backups stands
on its own. Also, I don't see backup and restore as adjacent. I see
them as thoroughly intertwined.

--
-David
david@pgmasters.net

On 11/27/18 8:56 AM, Simon Riggs wrote:

On Tue, 27 Nov 2018 at 03:13, David Steele <david@pgmasters.net
<mailto:david@pgmasters.net>> wrote:
 

The deprecated exclusive mode promises to make a difficult problem
simple but doesn't live up to that promise. That's why it was replaced
externally in 9.6 and why pg_basebackup has not used exclusive backups
since it was introduced in 9.2.

Non-exclusive backups have been available since 9.6 and several
third-party solutions support this mode, in addition to pg_basebackup.

The recently introduced recovery changes will break current automated
solutions so this seems like a good opportunity to make improvements on
the backup side as well.

I'll submit a patch for the 2019-01 commitfest.

-1 for removal, in this release

It's not there because anyone likes it, it's there because removing it
is a risk

Recent changes are the reason to keep it, not a reason to remove.

How so?

--
-David
david@pgmasters.net