Possible race condition in pg_basebackup
Hello,
We are encountering a possible race condition when executing several
`pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
while initialising streaming replicas. Our automation starts 3
pg_basebackup processes on 3 future replica servers within 1 second of each
other. One of them almost always fails with "requested WAL segment ... has
already been removed".
wal_keep_size = 0 (the default) on the server.
The server log shows three checkpoints starting within 1 second of each
other, recycling the WAL segment needed by the failing pg_basebackup.
Looking at the pg_basebackup code, we can see that it sends the BASE_BACKUP
command to the server, triggering the checkpoint, before
calling StartLogStreamer() that creates the slot and begins streaming WAL.
The time gap between the BASE_BACKUP and the slot creation seems to be the
reason for the failure we see.
Two obvious workarounds exist: create slots as a separate step before
running pg_basebackup, or update wal_keep_size to some sensible value to
prevent WAL segments from being recycled prematurely.
It seems to me, however, that pg_basebackup can be improved, regardless of
the existing workarounds, to create the slot, if it's being requested,
before starting the actual backup.
I'd like to hear feedback from the community on this.
Thanks
Nick Ivanov
EDB
On 21 Aug 2026, at 13:30, Nick Ivanov <nick.ivanov@enterprisedb.com> wrote:
Hi Nick,
I think your analysis is correct. More precisely, the problematic
sequence seems to be:
1. The first BASE_BACKUP takes its starting checkpoint and returns its
REDO location as xlogstart.
2. There is no slot protecting xlogstart yet.
3. Another checkpoint occurs before the slot is created. In your test it
is forced by another BASE_BACKUP, but it could also be a scheduled or
manually requested checkpoint.
4. That checkpoint can recycle the segment containing xlogstart.
5. Only then does the first pg_basebackup enter StartLogStreamer(), create
its slot with RESERVE_WAL, and request WAL starting at xlogstart.
RESERVE_WAL can only reserve WAL which still exists when the slot is
created. It cannot retrospectively protect the earlier base-backup start
point. The active-backup counter established by pg_backup_start() makes
WAL contain the required full-page images; it does not retain WAL
segments.
Interestingly, essentially this race was noticed during the 2017 review
of --create-slot. Jeff Janes asked whether the slot had to be created
before the checkpoint because streaming starts at the checkpoint's REDO
location [0]/messages/by-id/1505248760.16872.1.camel@credativ.de. It was thought that a subsequent checkpoint before the WAL
streamer connected was unlikely enough that the concern might be only
theoretical [1]/messages/by-id/CAMkU=1z_OucAFdersKxqjKsNUwFCrjvqV0=d-5o9Q17nR53s_g@mail.gmail.com. Your concurrent backups make that subsequent-checkpoint
scenario easy to hit, but the identity of whoever requested the checkpoint
is not important. A base backup must not depend on no checkpoint occurring
in this interval.
There is also a mismatch with the current documentation, which says that
--create-slot creates the slot "before starting the backup" [2]https://www.postgresql.org/docs/current/app-pgbasebackup.html. In the
current code, pg_basebackup sends BASE_BACKUP and obtains xlogstart before
StartLogStreamer() creates the slot.
I tried a small client-side proof-of-concept which creates the slot before
BASE_BACKUP. It closes this window, but leaves the slot inactive during
the starting checkpoint, which interacts badly with
idle_replication_slot_timeout. It also does not protect an existing slot
whose restart_lsn is still NULL. Therefore, I think this needs a design
discussion rather than just moving client-side calls around.
Perhaps BASE_BACKUP should retain WAL from the exact REDO location of the
checkpoint it selected until the WAL streamer has taken over. This also
handles backups from a standby, where the selected restartpoint may be
older than the current replay location. The server knows the exact LSN
and can release such a backup-owned retention horizon on backup completion
or abort, without transferring a temporary slot between the two client
connections.
Pre-creating the physical slots with WAL reserved is a reliable workaround.
A positive wal_keep_size also makes the window less likely, but is a
size-based cushion rather than protection tied to these backups.
As an operational aside, when provisioning several replicas at once I
would normally store one reusable base backup with WAL-G or pgBackRest and
restore it several times. That avoids making the primary transmit the
same cluster and take several backup checkpoints. But this does not make
the pg_basebackup race acceptable; I think you found a real bug.
Thank you!
Best regards, Andrey Borodin.
[0]: /messages/by-id/1505248760.16872.1.camel@credativ.de
[1]: /messages/by-id/CAMkU=1z_OucAFdersKxqjKsNUwFCrjvqV0=d-5o9Q17nR53s_g@mail.gmail.com
[2]: https://www.postgresql.org/docs/current/app-pgbasebackup.html
Hello Nick
On 2026-Aug-21, Nick Ivanov wrote:
We are encountering a possible race condition when executing several
`pg_basebackup --wal-method=stream --slot=... --create-slot` concurrently
while initialising streaming replicas. Our automation starts 3
pg_basebackup processes on 3 future replica servers within 1 second of each
other. One of them almost always fails with "requested WAL segment ... has
already been removed".
I think this is related to this thread here:
/messages/by-id/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com
and to this commit
Author: Amit Kapila <akapila@postgresql.org>
Branch: master Release: REL_19_BR [006dd4b2e] 2025-12-08 05:21:22 +0000
Branch: REL_18_STABLE Release: REL_18_2 [d3ceb2084] 2025-12-08 05:33:14 +0000
Prevent invalidation of newly created replication slots.
A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.
Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.
The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation to serialize WAL reservation and checkpoint's
minimum restart_lsn computation. This ensures that, if WAL reservation
occurs first, the checkpoint waits until restart_lsn is updated before
removing WAL. If the checkpoint runs first, subsequent WAL reservations
pick a position at or after the latest checkpoint's redo pointer.
We can't use the same fix for branch 17 and prior because commit
2090edc6f3 changed to compute to the minimum restart_LSN among slot's at
the beginning of checkpoint (or restart point). The fix for 17 and prior
branches is under discussion and will be committed separately.
Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
Author: Hou Zhijie <houzj.fnst@fujitsu.com>
Reviewed-by: Vitaly Davydov <v.davydov@postgrespro.ru>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 18
Discussion: /messages/by-id/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com
and to this other commit
Author: Amit Kapila <akapila@postgresql.org>
Branch: REL_17_STABLE Release: REL_17_8 [3510ebeb0] 2026-01-08 07:17:56 +0000
Branch: REL_16_STABLE Release: REL_16_12 [24cce33c3] 2026-01-08 07:07:23 +0000
Branch: REL_15_STABLE Release: REL_15_16 [aae05622a] 2026-01-08 06:54:52 +0000
Branch: REL_14_STABLE Release: REL_14_21 [7406df605] 2026-01-08 06:44:28 +0000
Prevent invalidation of newly created replication slots.
A race condition could cause a newly created replication slot to become
invalidated between WAL reservation and a checkpoint.
Previously, if the required WAL was removed, we retried the reservation
process. However, the slot could still be invalidated before the retry if
the WAL was not yet removed but the checkpoint advanced the redo pointer
beyond the slot's intended restart LSN and computed the minimum LSN that
needs to be preserved for the slots.
The fix is to acquire an exclusive lock on ReplicationSlotAllocationLock
during WAL reservation, and a shared lock during the minimum LSN
calculation at checkpoints to serialize the process. This ensures that, if
WAL reservation occurs first, the checkpoint waits until restart_lsn is
updated before calculating the minimum LSN. If the checkpoint runs first,
subsequent WAL reservations pick a position at or after the latest
checkpoint's redo pointer.
We used a similar fix in HEAD (via commit 006dd4b2e5) and 18. The
difference is that in 17 and prior branches we need to additionally handle
the race condition with slot's minimum LSN computation during checkpoints.
Reported-by: suyu.cmj <mengjuan.cmj@alibaba-inc.com>
Author: Hou Zhijie <houzj.fnst@fujitsu.com>
Author: vignesh C <vignesh21@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Backpatch-through: 14
Discussion: /messages/by-id/5e045179-236f-4f8f-84f1-0f2566ba784c.mengjuan.cmj@alibaba-inc.com
What version are you using?
If you're using a version that contains these fixes, then we may have
some slightly different bug ...
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No hay ausente sin culpa ni presente sin disculpa" (Prov. francés)