behave of --create-slot option

2018-05-28 16:41 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>:

I am writing a article about PostgreSQL 11 features. Now I am looking on new
option --create-slot option of pg_basebackup command.

I don't understand to use case for this option, because It fails when
requested slot already exists. I cannot to imagine use case for this. If I
write some scripts, then I prefer the behave like "create if doesn't exist
or do nothing".

In prior versions, you should create a slot (via SQL function or
streaming replication protocol) *before* run pg_basebackup. In 11, use
--create-slot option and you don't need an extra step (of course, you
should drop that slot after the end of backup -- if that is not a new
standby. It also does not make sense to use _persistent_ replication
slots for backup because you are probably archiving WAL). IMHO the use
case is new standbys that will use replication slots.

Any repeated running of script with this option should to fail. Is it
required? Why?

As I said, you should have an extra step to drop that slot (even
before 11). There is no "create replication slots if not exists". If
you are repeatedly running a script, why don't you let pg_basebackup
use temporary replication slots?

--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

On 29 May 2018 at 03:41, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Hi

I am writing a article about PostgreSQL 11 features. Now I am looking on
new option --create-slot option of pg_basebackup command.

I don't understand to use case for this option, because It fails when
requested slot already exists. I cannot to imagine use case for this. If I
write some scripts, then I prefer the behave like "create if doesn't exist
or do nothing".

Any repeated running of script with this option should to fail. Is it
required? Why?

The slot is intended to then be used by a replica that was created by
pg_basebackup. I think it writes the slot name into recovery.conf; if it
doesn't, it should.

So you use a unique slot name for each replica.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

2018-05-29 3:28 GMT+02:00 Craig Ringer <craig@2ndquadrant.com>:

On 29 May 2018 at 03:41, Pavel Stehule <pavel.stehule@gmail.com> wrote:

Hi

I am writing a article about PostgreSQL 11 features. Now I am looking on
new option --create-slot option of pg_basebackup command.

I don't understand to use case for this option, because It fails when
requested slot already exists. I cannot to imagine use case for this. If I
write some scripts, then I prefer the behave like "create if doesn't exist
or do nothing".

Any repeated running of script with this option should to fail. Is it
required? Why?

The slot is intended to then be used by a replica that was created by
pg_basebackup. I think it writes the slot name into recovery.conf; if it
doesn't, it should.

So you use a unique slot name for each replica.

I understand so slot should be unique. But same result (unique rep slot)
can be done, if it does nothing when slot exists already. This behave is
not idempotent.

Maybe I am search problem, where it is not. Just, when I see some "create
object" option, I expect any way, how I can enforce "--if-exists", because
it was necessary in major cases.

Regards

Pavel

On 29 May 2018 at 11:51, Pavel Stehule <pavel.stehule@gmail.com> wrote:

I understand so slot should be unique. But same result (unique rep slot)
can be done, if it does nothing when slot exists already. This behave is
not idempotent.

Maybe I am search problem, where it is not. Just, when I see some "create
object" option, I expect any way, how I can enforce "--if-exists", because
it was necessary in major cases.

If the slot already exists, don't you expect it to be in use for an
existing replica?

I guess what you seem to want is to be able to delete the replica then
re-clone it and use the same slot?

Generally I'd expect you to delete the slot when you remove the replica.
Really what this says to me is that we should have a way to delete the
upstream slot when we promote or decommission a physical replica.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

2018-05-29 6:11 GMT+02:00 Craig Ringer <craig@2ndquadrant.com>:

On 29 May 2018 at 11:51, Pavel Stehule <pavel.stehule@gmail.com> wrote:

I understand so slot should be unique. But same result (unique rep slot)
can be done, if it does nothing when slot exists already. This behave is
not idempotent.

Maybe I am search problem, where it is not. Just, when I see some "create
object" option, I expect any way, how I can enforce "--if-exists", because
it was necessary in major cases.

If the slot already exists, don't you expect it to be in use for an
existing replica?

the slot name is unique, so I don't expect it - when I use some name from
script

I guess what you seem to want is to be able to delete the replica then
re-clone it and use the same slot?

maybe. Now, it looks "asymmetric"

Generally I'd expect you to delete the slot when you remove the replica.
Really what this says to me is that we should have a way to delete the
upstream slot when we promote or decommission a physical replica.

When I think about this option and designed behave, I am inclined to think
so it can be hard to use, and better create slot outside, and outside drop
slot too.

I hope so I understand to motivation for this option - but I see issues:

1. pg_basebackup can fail from some other reasons, but there is not special
status for this issue.
2. when I use this option in any script, then I should to remove possibly
exists slot before, to minimize risk of fail of this command.

I understand, current behave can be wanted like protection against unwanted
running some deployment script. But can be problematic too, when
pg_basebackup or or some other fails from unexpected reasons.

Regards

Pavel

2018-05-29 1:31 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>:

I hope so I understand to motivation for this option - but I see issues:

1. pg_basebackup can fail from some other reasons, but there is not special
status for this issue.
2. when I use this option in any script, then I should to remove possibly
exists slot before, to minimize risk of fail of this command.

If pg_basebackup failed for some other reason *after* the replication
slot was created (say, permission problem) then we should try to
cleanup the old slot. That should be the behavior if we want to try to
be idempotent ("try" because if we have a network problem it won't be
possible to remove the replication slot).

--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

2018-05-29 16:53 GMT+02:00 Euler Taveira <euler@timbira.com.br>:

2018-05-29 1:31 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>:

I hope so I understand to motivation for this option - but I see issues:

1. pg_basebackup can fail from some other reasons, but there is not

special

status for this issue.
2. when I use this option in any script, then I should to remove possibly
exists slot before, to minimize risk of fail of this command.

If pg_basebackup failed for some other reason *after* the replication
slot was created (say, permission problem) then we should try to
cleanup the old slot. That should be the behavior if we want to try to
be idempotent ("try" because if we have a network problem it won't be
possible to remove the replication slot).

It has sense for me

Pavel

On Tue, May 29, 2018 at 09:21:00PM +0200, Pavel Stehule wrote:

2018-05-29 16:53 GMT+02:00 Euler Taveira <euler@timbira.com.br>:

If pg_basebackup failed for some other reason *after* the replication
slot was created (say, permission problem) then we should try to
cleanup the old slot. That should be the behavior if we want to try to
be idempotent ("try" because if we have a network problem it won't be
possible to remove the replication slot).

It has sense for me

Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is defined
by the caller when --create-slot is used. Do we want an open item for
this issue?
--
Michael

On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote:

Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is defined
by the caller when --create-slot is used. Do we want an open item for
this issue?

Sounds like new development to me. This isn't a bug.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote:

On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote:

Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is defined
by the caller when --create-slot is used. Do we want an open item for
this issue?

Sounds like new development to me. This isn't a bug.

Still, it seems to me that the user experience is a bit horrible with
this new interface of pg_basebackup. If --create-slot is used, then a
slot is created before starting a backup. If the slot already exists,
then pg_basebackup complains and exits. In order to drop the slot with
a only user who has replication access rights (because nobody is really
going to have a user who has SQL access so as the slot is dropped), then
the only simple way is to use pg_receivewal --drop-slot, making the
whole flow inconsistent. pg_basebackup is usually in server-side
packages, and pg_receivewal is on the client side. The server packages
requiring the client packages, then we are sure that pg_basebackup will
drag in pg_receivewal.

Still, shouldn't there be a --drop-slot option in pg_basebackup? In
this case, if --drop-slot is used, then the slot is dropped and
pg_basebackup exits immediately.
--
Michael

2018-06-01 9:00 GMT-03:00 Michael Paquier <michael@paquier.xyz>:

On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote:

On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote:

Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is defined
by the caller when --create-slot is used. Do we want an open item for
this issue?

Sounds like new development to me. This isn't a bug.

Still, it seems to me that the user experience is a bit horrible with
this new interface of pg_basebackup. If --create-slot is used, then a
slot is created before starting a backup. If the slot already exists,
then pg_basebackup complains and exits. In order to drop the slot with
a only user who has replication access rights (because nobody is really
going to have a user who has SQL access so as the slot is dropped), then
the only simple way is to use pg_receivewal --drop-slot, making the
whole flow inconsistent. pg_basebackup is usually in server-side
packages, and pg_receivewal is on the client side. The server packages
requiring the client packages, then we are sure that pg_basebackup will
drag in pg_receivewal.

Debian and derivatives put pg_basebackup in the client package
(indeed, it a client program). A possible fix is to drop the slot if
there is an error. However, if the problem is connectivity, we can't
drop it. In this case, we can print a user-friendly error saying that
the user should drop that slot before try again). If we follow this
idea, then I consider it to be a bug fix.

Still, shouldn't there be a --drop-slot option in pg_basebackup? In
this case, if --drop-slot is used, then the slot is dropped and
pg_basebackup exits immediately.

I don't like it. You should send an extra command to recover from an
error. It also does not follow the idempotent behavior. I know that
you are trying to mimic pg_receivewal options but I prefer another
option that removes the slot if it exists (say,
--drop-slot-if-exists). In this case, I consider this option as new
development.

--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

2018-06-01 18:36 GMT+02:00 Euler Taveira <euler@timbira.com.br>:

2018-06-01 9:00 GMT-03:00 Michael Paquier <michael@paquier.xyz>:

On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote:

On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz>

wrote:

Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is

defined

by the caller when --create-slot is used. Do we want an open item for
this issue?

Sounds like new development to me. This isn't a bug.

Still, it seems to me that the user experience is a bit horrible with
this new interface of pg_basebackup. If --create-slot is used, then a
slot is created before starting a backup. If the slot already exists,
then pg_basebackup complains and exits. In order to drop the slot with
a only user who has replication access rights (because nobody is really
going to have a user who has SQL access so as the slot is dropped), then
the only simple way is to use pg_receivewal --drop-slot, making the
whole flow inconsistent. pg_basebackup is usually in server-side
packages, and pg_receivewal is on the client side. The server packages
requiring the client packages, then we are sure that pg_basebackup will
drag in pg_receivewal.

Debian and derivatives put pg_basebackup in the client package
(indeed, it a client program). A possible fix is to drop the slot if
there is an error. However, if the problem is connectivity, we can't
drop it. In this case, we can print a user-friendly error saying that
the user should drop that slot before try again). If we follow this
idea, then I consider it to be a bug fix.

Still, shouldn't there be a --drop-slot option in pg_basebackup? In
this case, if --drop-slot is used, then the slot is dropped and
pg_basebackup exits immediately.

I don't like it. You should send an extra command to recover from an
error. It also does not follow the idempotent behavior. I know that
you are trying to mimic pg_receivewal options but I prefer another
option that removes the slot if it exists (say,
--drop-slot-if-exists). In this case, I consider this option as new
development.

+1

Pavel