Stale description for pg_basebackup

At Tue, 20 Apr 2021 13:32:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

Hello.

It seems to me that there's a stale description in the documentation
of pg_basebackup.

https://www.postgresql.org/docs/13/app-pgbasebackup.html

Note that there are some limitations in taking a backup from a standby:

...

If you are using -X none, there is no guarantee that all WAL files
required for the backup are archived at the end of backup.

Actually, pg_basebackup waits for the all required files to be
archived, which is an established behavior by commit
52f8a59dd9@PG10. However, the same commit seems to have forgot to
change the doc for pg_basebackup. (The current description is
introduced by 9a4d51077c@PG10)

The attached is a proposal to rewrite it as the following.

+ If you are using -X none, pg_basebackup may wait for a long time for
+ all the required WAL files to be archived. In that case, You may need
+ to call pg_switch_wal() on the primary to complete it sooner.

I forgot to preserve the description about *primary*. It should be as
the following instead.

+ If you are using -X none, there is no guarantee on the primary that
+ all WAL files required for the backup are archived at the end of
+ backup. When the standby is configured as archive_mode=always,
+ pg_basebackup may wait for a long time for all the required WAL files
+ to be archived. In that case, You may need to call pg_switch_wal() on
+ the primary to complete it sooner.

Attached.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

At Wed, 21 Apr 2021 10:43:30 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

At Tue, 20 Apr 2021 13:32:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

Hello.

It seems to me that there's a stale description in the documentation
of pg_basebackup.

https://www.postgresql.org/docs/13/app-pgbasebackup.html

Note that there are some limitations in taking a backup from a standby:

...

If you are using -X none, there is no guarantee that all WAL files
required for the backup are archived at the end of backup.

Actually, pg_basebackup waits for the all required files to be
archived, which is an established behavior by commit
52f8a59dd9@PG10. However, the same commit seems to have forgot to
change the doc for pg_basebackup. (The current description is
introduced by 9a4d51077c@PG10)

The attached is a proposal to rewrite it as the following.

+ If you are using -X none, pg_basebackup may wait for a long time for
+ all the required WAL files to be archived. In that case, You may need
+ to call pg_switch_wal() on the primary to complete it sooner.

I forgot to preserve the description about *primary*. It should be as
the following instead.

+ If you are using -X none, there is no guarantee on the primary that
+ all WAL files required for the backup are archived at the end of
+ backup. When the standby is configured as archive_mode=always,
+ pg_basebackup may wait for a long time for all the required WAL files
+ to be archived. In that case, You may need to call pg_switch_wal() on
+ the primary to complete it sooner.

Hmm. Some words need to be qualified. Attached.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021/04/21 11:09, Kyotaro Horiguchi wrote:

At Wed, 21 Apr 2021 10:43:30 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

At Tue, 20 Apr 2021 13:32:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in

Hello.

It seems to me that there's a stale description in the documentation
of pg_basebackup.

I think you're right.

Hmm. Some words need to be qualified. Attached.

+       If you are using <literal>-X none</literal>, there is no guarantee on
+       the primary that all WAL files required for the backup are archived at
+       the end of backup.

I don't think that this should be picked up as a limitation of standby backup.
Because users basically want to make pg_basebackup wait for all required
WAL files to be archived on the standby, in the standby backup case.

When <varname>archive_mode</varname> is set
+ to <literal>on</literal> on the

"on" should be "always"?

+       standby, <application>pg_basebackup</application> may wait for a long
+       time for all the required WAL files to be archived.  In that case, You
+       may need to call <function>pg_switch_wal()</function> on the primary to
+       complete it sooner.

What about the following description?

-------------------
When you are using -X none, if write activity on the primary is low,
pg_basebackup may need to wait a long time for all WAL files required for
the backup to be archived. It may be useful to run pg_switch_wal
on the primary in order to trigger an immediate WAL file switch and archiving.
-------------------

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

Ugg. I was confused.

At Wed, 21 Apr 2021 23:06:56 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in

Hmm. Some words need to be qualified. Attached.

+ If you are using <literal>-X none</literal>, there is no guarantee
on
+ the primary that all WAL files required for the backup are archived
at
+       the end of backup.

I don't think that this should be picked up as a limitation of standby
backup.
Because users basically want to make pg_basebackup wait for all
required
WAL files to be archived on the standby, in the standby backup case.

Yeah, you're right. I think it is what I thought at first. The last
proposal is a result of some confusion..

When <varname>archive_mode</varname> is set
+ to <literal>on</literal> on the

"on" should be "always"?

Yes..

+ standby, <application>pg_basebackup</application> may wait for a
long
+ time for all the required WAL files to be archived.  In that case,
You
+ may need to call <function>pg_switch_wal()</function> on the primary
to
+       complete it sooner.

What about the following description?

-------------------
When you are using -X none, if write activity on the primary is low,
pg_basebackup may need to wait a long time for all WAL files required
for
the backup to be archived. It may be useful to run pg_switch_wal
on the primary in order to trigger an immediate WAL file switch and
archiving.
-------------------

Looks far better.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021/04/22 9:25, Kyotaro Horiguchi wrote:

What about the following description?

-------------------
When you are using -X none, if write activity on the primary is low,
pg_basebackup may need to wait a long time for all WAL files required
for
the backup to be archived. It may be useful to run pg_switch_wal
on the primary in order to trigger an immediate WAL file switch and
archiving.
-------------------

Looks far better.

Patch attached. I appended the following description to assist
users to understand why pg_basebackup may need wait a long time
when write activity is low on the primary.

------------------
pg_basebackup cannot force the standby to switch to
a new WAL file at the end of backup.
------------------

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

At Thu, 22 Apr 2021 10:56:10 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in

On 2021/04/22 9:25, Kyotaro Horiguchi wrote:

What about the following description?

-------------------
When you are using -X none, if write activity on the primary is low,
pg_basebackup may need to wait a long time for all WAL files required
for
the backup to be archived. It may be useful to run pg_switch_wal
on the primary in order to trigger an immediate WAL file switch and
archiving.
-------------------

Looks far better.

Patch attached. I appended the following description to assist
users to understand why pg_basebackup may need wait a long time
when write activity is low on the primary.

------------------
pg_basebackup cannot force the standby to switch to
a new WAL file at the end of backup.
------------------

I'm not sure which is the convention here, but I saw that some
function names in the doc are followed by parentheses (ie
pg_switch_wal()).

(prepended?) It seems a bit redundant but also a bit clearer. How
about the following simplification?

- It may be useful to run pg_switch_wal on the primary in order to
- trigger an immediate WAL file switch and archiving.
+ It may be useful to run pg_switch_wal() on the primary in that case.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021/04/22 11:19, Kyotaro Horiguchi wrote:

At Thu, 22 Apr 2021 10:56:10 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in

On 2021/04/22 9:25, Kyotaro Horiguchi wrote:

What about the following description?

-------------------
When you are using -X none, if write activity on the primary is low,
pg_basebackup may need to wait a long time for all WAL files required
for
the backup to be archived. It may be useful to run pg_switch_wal
on the primary in order to trigger an immediate WAL file switch and
archiving.
-------------------

Looks far better.

Patch attached. I appended the following description to assist
users to understand why pg_basebackup may need wait a long time
when write activity is low on the primary.

------------------
pg_basebackup cannot force the standby to switch to
a new WAL file at the end of backup.
------------------

I'm not sure which is the convention here, but I saw that some
function names in the doc are followed by parentheses (ie
pg_switch_wal()).

Either works for me. I didn't add "()" because I just used the same description
as that in func.sgml.

it may be useful to run <function>pg_switch_wal</function> on the
primary in order to trigger an immediate segment switch.)

(prepended?) It seems a bit redundant but also a bit clearer. How
about the following simplification?

- It may be useful to run pg_switch_wal on the primary in order to
- trigger an immediate WAL file switch and archiving.
+ It may be useful to run pg_switch_wal() on the primary in that case.

IMO "in order to..." part is helpful for us to understand why pg_switch_wal
is useful in this case. So I'd like to leave it.

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

At Thu, 22 Apr 2021 13:06:50 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in

Either works for me. I didn't add "()" because I just used the same
description
as that in func.sgml.

it may be useful to run <function>pg_switch_wal</function> on the
primary in order to trigger an immediate segment switch.)

..

IMO "in order to..." part is helpful for us to understand why
pg_switch_wal
is useful in this case. So I'd like to leave it.

Ok, I'm fine with both of them. Thanks for the explanation.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On 2021/04/22 13:25, Kyotaro Horiguchi wrote:

At Thu, 22 Apr 2021 13:06:50 +0900, Fujii Masao <masao.fujii@oss.nttdata.com> wrote in

Either works for me. I didn't add "()" because I just used the same
description
as that in func.sgml.

it may be useful to run <function>pg_switch_wal</function> on the
primary in order to trigger an immediate segment switch.)

..

IMO "in order to..." part is helpful for us to understand why
pg_switch_wal
is useful in this case. So I'd like to leave it.

Ok, I'm fine with both of them. Thanks for the explanation.

Pushed. Thanks!

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION