Timing issue in pg_basebackup
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254274psql -h localhost -U postgresBuilt from patchset v1 (message #1), October 19, 2026 at 02:46 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t254274_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t254274_1 && git checkout t254274_1Patchset v1 (message #1) is on t254274_1
Hey!
I recently came across this issue when running pg_basebackup from 2
nodes at the same time through ansible. One of the nodes sometimes
reported: "requested WAL segment ... has already been removed". Tested
on v16, v17 and v18. The workaround was running them in sequence or
pause between runs.
It looks like there's a timing issue here:
1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
2. Backup A hasn't reserved that WAL yet.
3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) -
deletes segment 2.
4. Backup A asks for segment 2, which is gone, and fails.
I've included a patch that moves the replication slot handing to run
before the backup part as a separate function.
Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for
example, --checkpoint=spread is used since the slot is open for longer.
* If any error arises in the backup part, now there will be a lingering
slot when previously no slot was created.
*** DISCLAIMER ***
I've used an LLM to help me write the test - I'm rubbish at Perl.
Love to get some feedback.
Regards,
Jesper St John
On Mon, 19 Oct 2026 at 19:33, Jesper St John <jesper@stjohn.se> wrote:
Hey!
I recently came across this issue when running pg_basebackup from 2
nodes at the same time through ansible. One of the nodes sometimes
reported: "requested WAL segment ... has already been removed". Tested
on v16, v17 and v18. The workaround was running them in sequence or
pause between runs.It looks like there's a timing issue here:
1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
2. Backup A hasn't reserved that WAL yet.
3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint) -
deletes segment 2.
4. Backup A asks for segment 2, which is gone, and fails.
Hi!
Looks like this is a real issue.
I've included a patch that moves the replication slot handing to run
before the backup part as a separate function.Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for
example, --checkpoint=spread is used since the slot is open for longer.
* If any error arises in the backup part, now there will be a lingering
slot when previously no slot was created.
lingering slots looks like a real regression here. can we remove them
when basebackup client disconnects?
NIT: comment in PrepareLogStreamer loses its meaning after code motion:
+ /* Get a second connection */
+ bgconn = GetConnection();
*** DISCLAIMER ***
I've used an LLM to help me write the test - I'm rubbish at Perl.
OK
--
Best regards,
Kirill Reshke
On 2026-10-20 23:11, Kirill Reshke wrote:
On Mon, 19 Oct 2026 at 19:33, Jesper St John <jesper@stjohn.se> wrote:
Hey!
I recently came across this issue when running pg_basebackup from 2
nodes at the same time through ansible. One of the nodes sometimes
reported: "requested WAL segment ... has already been removed". Tested
on v16, v17 and v18. The workaround was running them in sequence or
pause between runs.It looks like there's a timing issue here:
1. Backup A starts and runs a checkpoint - needs WAL from segment 2.
2. Backup A hasn't reserved that WAL yet.
3. Backup B starts and runs a checkpoint (or anyone runs a checkpoint)
-
deletes segment 2.
4. Backup A asks for segment 2, which is gone, and fails.Hi!
Looks like this is a real issue.
Thanks for looking into it.
I've included a patch that moves the replication slot handing to run
before the backup part as a separate function.Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for
example, --checkpoint=spread is used since the slot is open for
longer.
* If any error arises in the backup part, now there will be a
lingering
slot when previously no slot was created.lingering slots looks like a real regression here. can we remove them
when basebackup client disconnects?
A lingering slot could also happen today - only with this patch there's
a higher risk of it happening.
Would some cleanup code in a function and use it with atexit() work?
NIT: comment in PrepareLogStreamer loses its meaning after code motion:
+ /* Get a second connection */
+ bgconn = GetConnection();
I've moved the code to its own function - the comment were there
already.
Show quoted text
*** DISCLAIMER ***
I've used an LLM to help me write the test - I'm rubbish at Perl.OK
On Wed, 21 Oct 2026 at 22:16, Jesper St John <jesper@stjohn.se> wrote:
On 2026-10-20 23:11, Kirill Reshke wrote:
I've included a patch that moves the replication slot handing to run
before the backup part as a separate function.Some possible caveats already:
* Since moving the slot creation, we might run in to trouble if for
example, --checkpoint=spread is used since the slot is open for
longer.
* If any error arises in the backup part, now there will be a
lingering
slot when previously no slot was created.lingering slots looks like a real regression here. can we remove them
when basebackup client disconnects?A lingering slot could also happen today - only with this patch there's
a higher risk of it happening.
Higher risk of lingering slots is also a regression. But looks like
this risk is anyway close to zero.
Would some cleanup code in a function and use it with atexit() work?
Something like this should do, can be made a separate patch thought
NIT: comment in PrepareLogStreamer loses its meaning after code motion:
+ /* Get a second connection */
+ bgconn = GetConnection();I've moved the code to its own function - the comment were there
already.
Yep, the point is It looks very out-of-context here. Like, the first
line of the function says we're taking a second connection. Second to
what (to basebackup)? Maybe rephrase here as 'Get a wal-serving
connection' or something.
--
Best regards,
Kirill Reshke