Fix orphaned 2pc file which may casue instance restart failed
Hi all, I found that there is a race condition between two global transaction, which may cause instance
restart failed with error 'could not access status of transaction xxx","Could not open file ""pg_xact/xxx"": No such file or directory'.
The scenery to reproduce the problem is:
1. gxact1 is doing `FinishPreparedTransaction` and checkpoint
is issued, so gxact1 will generate a 2pc file.
2. then gxact1 was removed from `TwoPhaseState->prepXacts` and
its state memory was returned to freelist.
3. but just before gxact1 remove its 2pc file, gxact2 is issued,
gxact2 will reuse the same state memory of gxact1 and will
reset `gxact->ondisk` to false.
4. gxact1 continue and found that `gxact->ondisk` is false, it won't
remove its 2pc file. This file is orphaned.
If gxact1's local xid is not frozen, the startup process will remove
the orphaned 2pc file. However, if the xid's corresponding clog file is
truncated by `vacuum`, the startup process will raise error 'could not
access status of transaction xxx', due to it could not found the
transaction's status file in dir `pg_xact`.
The potential fix is attached.
Attachments:
0001-Fix-orphaned-2pc-file-which-may-casue-instance-resta.patchapplication/octet-stream; charset=gb18030; name=0001-Fix-orphaned-2pc-file-which-may-casue-instance-resta.patchDownload
From 0230da90f7d930422db388353a4040ee4afd6112 Mon Sep 17 00:00:00 2001
From: wuchengwen <drec.wu@foxmail.com>
Date: Sun, 8 Sep 2024 12:09:12 +0800
Subject: [PATCH] Fix orphaned 2pc file which may casue instance restart failed
There is a race condition for global transaction when removing
2pc files:
1. gxact1 is doing `FinishPreparedTransaction` and checkpoint
is issued, so gxact1 will generate a 2pc file.
2. then gxact1 was removed from `TwoPhaseState->prepXacts` and
was put into freelist.
3. but just before gxact1 remove its 2pc file, gxact2 is issued,
gxact2 will reuse the same state memory of gxact1 and will
reset `gxact->ondisk` to false.
4. gxact1 continue and found that `gxact->ondisk` is false, it won't
remove its 2pc file. This file is orphaned.
If gxact1's local xid is not frozen, the startup process will remove
the orphaned 2pc file. However, if the clog file is truncated by `vacuum`,
the startup process will raise error 'could not access status of transaction xxx',
due to it could not found the transaction's status file in dir `pg_xact`.
---
src/backend/access/transam/twophase.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index e98286d768..db8d95f8cb 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -1505,6 +1505,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
GlobalTransaction gxact;
PGPROC *proc;
TransactionId xid;
+ bool ondisk;
char *buf;
char *bufptr;
TwoPhaseFileHeader *hdr;
@@ -1657,6 +1658,8 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
PredicateLockTwoPhaseFinish(xid, isCommit);
+ ondisk = gxact->ondisk;
+
/* Clear shared memory state */
RemoveGXact(gxact);
@@ -1672,7 +1675,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
/*
* And now we can clean up any files we may have left.
*/
- if (gxact->ondisk)
+ if (ondisk)
RemoveTwoPhaseFile(xid, true);
MyLockedGxact = NULL;
--
2.39.2 (Apple Git-143)
On Sun, Sep 08, 2024 at 01:01:37PM +0800, 清浅 wrote:
Hi all, I found that there is a race condition
between two global transaction, which may cause instance restart
failed with error 'could not access status of transaction
xxx","Could not open file ""pg_xact/xxx"": No such file or
directory'.The scenery to reproduce the problem is:
1. gxact1 is doing `FinishPreparedTransaction` and checkpoint
is issued, so gxact1 will generate a 2pc file.
2. then gxact1 was removed from `TwoPhaseState->prepXacts` and
its state memory was returned to freelist.
3. but just before gxact1 remove its 2pc file, gxact2 is issued,
gxact2 will reuse the same state memory of gxact1 and will
reset `gxact->ondisk` to false.
4. gxact1 continue and found that `gxact->ondisk` is false, it won't
remove its 2pc file. This file is orphaned.If gxact1's local xid is not frozen, the startup process will remove
the orphaned 2pc file. However, if the xid's corresponding clog file is
truncated by `vacuum`, the startup process will raise error 'could not
access status of transaction xxx', due to it could not found the
transaction's status file in dir `pg_xact`.
Hmm. I've not seen that in the field. Let me check that..
--
Michael
On Wed, Sep 11, 2024 at 06:21:37PM +0900, Michael Paquier wrote:
If gxact1's local xid is not frozen, the startup process will remove
the orphaned 2pc file. However, if the xid's corresponding clog file is
truncated by `vacuum`, the startup process will raise error 'could not
access status of transaction xxx', due to it could not found the
transaction's status file in dir `pg_xact`.Hmm. I've not seen that in the field. Let me check that..
Ah, yes, that's clearly broken. The origin of the problem was
effe7d9552dd, which was from me, so backpatched all the way down after
adding a comment explaining the reason why we'd better read the value
before the 2PC state lock is released.
--
Michael