Race between prepared transaction commit and checkpointer
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253758psql -h localhost -U postgresBuilt from patchset v5 (message #5), September 20, 2026 at 11:32 AM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253758_5 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253758_5 && git checkout t253758_5Patchset v5 (message #5) is on t253758_5
Hi, hackers!
I think I found an issue in two-phase commit code, I've attached patch
with reproducing, pls take a look(master rev 9f4bd91a196).
I've got coredump with backtrace like this
#0 __pthread_kill_implementation (no_tid=0, signo=6,
threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at
./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
at ./nptl/pthread_kill.c:89
#3 0x00007de7ef24527e in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4 0x00007de7ef2288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000610d848485bb in ExceptionalCondition
(conditionName=0x610d84910812 "XLogRecPtrIsValid(RecPtr)",
fileName=0x610d849107c2 "xlogreader.c", lineNumber=240) at assert.c:65
#6 0x0000610d840b0926 in XLogBeginRead (state=0x610d948b06a8, RecPtr=0)
at xlogreader.c:240
#7 0x0000610d8408ba9a in XlogReadTwoPhaseData (lsn=0,
buf=0x7fff8c941248, len=0x0) at twophase.c:1437
#8 0x0000610d8408be05 in FinishPreparedTransaction (gid=0x610d947e6698
"test_transaction", isCommit=true) at twophase.c:1544
#9 0x0000610d8462b70e in standard_ProcessUtility (pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:655
#10 0x0000610d8462ade6 in ProcessUtility (pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:525
#11 0x0000610d846293ee in PortalRunUtility (portal=0x610d94864a20,
pstmt=0x610d947e6798, isTopLevel=true, setHoldSnapshot=false,
dest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1149
#12 0x0000610d84629668 in PortalRunMulti (portal=0x610d94864a20,
isTopLevel=true, setHoldSnapshot=false, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1307
#13 0x0000610d84628ac0 in PortalRun (portal=0x610d94864a20,
count=9223372036854775807, isTopLevel=true, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730) at pquery.c:784
#14 0x0000610d84620b10 in exec_simple_query (query_string=0x610d947e5cb0
"COMMIT PREPARED 'test_transaction';") at postgres.c:1297
....
It seems we have race between prepared transaction commit and
checkpointer and it is old problem. We need TwoPhaseStateLock wrap the
reading state file in FinishPreparedTransaction code, the draft patch is
attached too, hope it will be helpful.
Best regards,
Maksim Melnikov
Hi Maskim,
Sure I'll look into it. I'll reproduce the bug in my machine amd I'll
review the patches attached in this email.
I'll let you know once I reproduce the issue in my machine and review the
patch.
With regards,
Osama Abdul Qader
On Thu, 10 Sept, 2026, 10:20 pm Maksim.Melnikov, <m.melnikov@postgrespro.ru>
wrote:
Show quoted text
Hi, hackers!
I think I found an issue in two-phase commit code, I've attached patch
with reproducing, pls take a look(master rev 9f4bd91a196).
I've got coredump with backtrace like this#0 __pthread_kill_implementation (no_tid=0, signo=6,
threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at
./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
at ./nptl/pthread_kill.c:89
#3 0x00007de7ef24527e in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4 0x00007de7ef2288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000610d848485bb in ExceptionalCondition
(conditionName=0x610d84910812 "XLogRecPtrIsValid(RecPtr)",
fileName=0x610d849107c2 "xlogreader.c", lineNumber=240) at assert.c:65
#6 0x0000610d840b0926 in XLogBeginRead (state=0x610d948b06a8, RecPtr=0)
at xlogreader.c:240
#7 0x0000610d8408ba9a in XlogReadTwoPhaseData (lsn=0,
buf=0x7fff8c941248, len=0x0) at twophase.c:1437
#8 0x0000610d8408be05 in FinishPreparedTransaction (gid=0x610d947e6698
"test_transaction", isCommit=true) at twophase.c:1544
#9 0x0000610d8462b70e in standard_ProcessUtility (pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:655
#10 0x0000610d8462ade6 in ProcessUtility (pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:525
#11 0x0000610d846293ee in PortalRunUtility (portal=0x610d94864a20,
pstmt=0x610d947e6798, isTopLevel=true, setHoldSnapshot=false,
dest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1149
#12 0x0000610d84629668 in PortalRunMulti (portal=0x610d94864a20,
isTopLevel=true, setHoldSnapshot=false, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1307
#13 0x0000610d84628ac0 in PortalRun (portal=0x610d94864a20,
count=9223372036854775807, isTopLevel=true, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730) at pquery.c:784
#14 0x0000610d84620b10 in exec_simple_query (query_string=0x610d947e5cb0
"COMMIT PREPARED 'test_transaction';") at postgres.c:1297
....It seems we have race between prepared transaction commit and
checkpointer and it is old problem. We need TwoPhaseStateLock wrap the
reading state file in FinishPreparedTransaction code, the draft patch is
attached too, hope it will be helpful.Best regards,
Maksim Melnikov
Thanks, I found one more suspicious place, pls take a look at
checkpointer CheckPointTwoPhase method, I am little bit suprised that
we acquire LW_SHARED lock here, because with this lock
we are changing gxact members, if some concurrent proc read this one
with LW_SHARED, it can get stale values. I am not sure but for example
it can be LookupGXact.
Anyway, if I am mistaken, sorry for noise.
Best regards,
Maksim Melnikov
Show quoted text
On 9/10/26 21:12, Osama Abdul Qader wrote:
Hi Maskim,
Sure I'll look into it. I'll reproduce the bug in my machine amd I'll
review the patches attached in this email.I'll let you know once I reproduce the issue in my machine and review
the patch.With regards,
Osama Abdul QaderOn Thu, 10 Sept, 2026, 10:20 pm Maksim.Melnikov,
<m.melnikov@postgrespro.ru> wrote:Hi, hackers!
I think I found an issue in two-phase commit code, I've attached
patch
with reproducing, pls take a look(master rev 9f4bd91a196).
I've got coredump with backtrace like this#0 __pthread_kill_implementation (no_tid=0, signo=6,
threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at
./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>,
signo=signo@entry=6)
at ./nptl/pthread_kill.c:89
#3 0x00007de7ef24527e in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#4 0x00007de7ef2288ff in __GI_abort () at ./stdlib/abort.c:79
#5 0x0000610d848485bb in ExceptionalCondition
(conditionName=0x610d84910812 "XLogRecPtrIsValid(RecPtr)",
fileName=0x610d849107c2 "xlogreader.c", lineNumber=240) at assert.c:65
#6 0x0000610d840b0926 in XLogBeginRead (state=0x610d948b06a8,
RecPtr=0)
at xlogreader.c:240
#7 0x0000610d8408ba9a in XlogReadTwoPhaseData (lsn=0,
buf=0x7fff8c941248, len=0x0) at twophase.c:1437
#8 0x0000610d8408be05 in FinishPreparedTransaction
(gid=0x610d947e6698
"test_transaction", isCommit=true) at twophase.c:1544
#9 0x0000610d8462b70e in standard_ProcessUtility
(pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:655
#10 0x0000610d8462ade6 in ProcessUtility (pstmt=0x610d947e6798,
queryString=0x610d947e5cb0 "COMMIT PREPARED 'test_transaction';",
readOnlyTree=false,
context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0,
dest=0x610d947e6b58, qc=0x7fff8c941730) at utility.c:525
#11 0x0000610d846293ee in PortalRunUtility (portal=0x610d94864a20,
pstmt=0x610d947e6798, isTopLevel=true, setHoldSnapshot=false,
dest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1149
#12 0x0000610d84629668 in PortalRunMulti (portal=0x610d94864a20,
isTopLevel=true, setHoldSnapshot=false, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730)
at pquery.c:1307
#13 0x0000610d84628ac0 in PortalRun (portal=0x610d94864a20,
count=9223372036854775807, isTopLevel=true, dest=0x610d947e6b58,
altdest=0x610d947e6b58, qc=0x7fff8c941730) at pquery.c:784
#14 0x0000610d84620b10 in exec_simple_query
(query_string=0x610d947e5cb0
"COMMIT PREPARED 'test_transaction';") at postgres.c:1297
....It seems we have race between prepared transaction commit and
checkpointer and it is old problem. We need TwoPhaseStateLock wrap
the
reading state file in FinishPreparedTransaction code, the draft
patch is
attached too, hope it will be helpful.Best regards,
Maksim Melnikov
Hi Maksim,
On 11 Sep 2026, Maksim Melnikov wrote:
I am not sure but for example
it can be LookupGXact.
Yes, your suspicion about LookupGXact() is correct. With your patch
applied, I could still trigger its Assert(gxact->prepare_start_lsn) by
pausing in the WAL branch and completing a checkpoint. Both holders
use LW_SHARED, so the checkpoint can proceed. This was a direct call
to the helper via a test wrapper, not an end-to-end logical replication
test.
I also reproduced the original race for both COMMIT PREPARED and
ROLLBACK PREPARED. Your fix blocks that interleaving in both cases.
Would it be better to take LW_EXCLUSIVE in CheckPointTwoPhase() and
protect the state reads in both FinishPreparedTransaction() and
LookupGXact() with LW_SHARED? That would address both readers without
serializing their I/O under a global exclusive lock. The reader lock
would still need to cover the WAL read, not just copying the LSN,
because the checkpoint can otherwise proceed to recycling that WAL.
Thank you!
Best regards, Andrey Borodin.
On 9/11/26 11:33, Andrey Borodin wrote:
Hi Maksim,
On 11 Sep 2026, Maksim Melnikov wrote:
I am not sure but for example
it can be LookupGXact.Yes, your suspicion about LookupGXact() is correct. With your patch
applied, I could still trigger its Assert(gxact->prepare_start_lsn) by
pausing in the WAL branch and completing a checkpoint. Both holders
use LW_SHARED, so the checkpoint can proceed. This was a direct call
to the helper via a test wrapper, not an end-to-end logical replication
test.I also reproduced the original race for both COMMIT PREPARED and
ROLLBACK PREPARED. Your fix blocks that interleaving in both cases.Would it be better to take LW_EXCLUSIVE in CheckPointTwoPhase() and
protect the state reads in both FinishPreparedTransaction() and
LookupGXact() with LW_SHARED? That would address both readers without
serializing their I/O under a global exclusive lock. The reader lock
would still need to cover the WAL read, not just copying the LSN,
because the checkpoint can otherwise proceed to recycling that WAL.Thank you!
Best regards, Andrey Borodin.
Hi Andrey,
Yes, I agree that we should acquire LW_EXCLUSIVE in CheckPointTwoPhase,
also I agree that better to wrap I/O on FinishPreparedTransaction with
just LW_SHARED, but I have some doubts about LookupGXact bacause as I
see here we change gxact state and it seems better do it with EXCLUSIVE
lock. Maybe we should acquire LW_EXCLUSIVE in LookupGXact and release it
and then acquire LW_SHARED for I/O. What do you think? For you
convenience I've attached draft patch.
Thanks you.
Best regards
Maksim Melnikov