pg_waldump: WAL in pg_wal.old/ inside a tar archive is read as WAL
Hi,
While testing the new PG 19 support for reading WAL from tar archives, I
found that member_is_wal_file() in src/bin/pg_waldump/archive_waldump.c
accepts WAL files from directories other than pg_wal/.
The comment says "Skip files in subdirectories other than pg_wal/", but the
check only compares the first six characters:
if (pathlen > XLOG_FNAME_LEN &&
strncmp(pathname, XLOGDIR, strlen(XLOGDIR)) != 0)
return false;
So pg_wal.old/000000010000000000000038 or
pg_wal/somedir/000000010000000000000038 is also taken as WAL. If such a
file comes first in the archive, the real pg_wal/ copy is then skipped as a
duplicate.
This also affects pg_verifybackup. With -Ft and no pg_wal.tar (for example
pg_basebackup -Xfetch), it runs pg_waldump on base.tar.
How I reproduced it on REL_19_STABLE:
1. Put a WAL segment from another cluster into the data directory as
pg_wal.old/000000010000000000000038, using the same name as the segment the
next backup starts in.
2. pg_basebackup -D bk -Ft -Xfetch. base.tar has
pg_wal.old/000000010000000000000038 as member 9 and
pg_wal/000000010000000000000038 as member 1004.
3. pg_waldump -p bk/base.tar -s 0/03800028 -e 0/03800128 exits 0 and prints
a COMMIT record from the other cluster. The real records at that LSN are
CHECKPOINT_REDO, RUNNING_XACTS, CHECKPOINT_ONLINE and BACKUP_END.
4. I zeroed the first records of the real pg_wal/000000010000000000000038
in base.tar. Without pg_wal.old/, pg_verifybackup -F t fails with "WAL
parsing failed for timeline 1". With pg_wal.old/ present, it says "backup
successfully verified".
The attached patch accepts a WAL file only at the top level of the archive
or directly in pg_wal/, and adds a test to
https://www.google.com/url?q=http://001_basic.pl&source=gmail&ust=1795274530565000&sa=E.
The test fails without the code change and passes with it.
With the patch, the corrupted backup above fails verification, and
pg_waldump prints the real records. I checked that pg_waldump output is
unchanged compared to reading the directory for tar files with segments in
order, reversed and shuffled, with pg_wal/ and ./ prefixes, with a
duplicate member, and with gzip, lz4 and zstd compression. The pg_waldump
and pg_verifybackup TAP suites pass on master and REL_19_STABLE.
I think this should be backpatched to 19.
Regards,
Sai Pranav
On Fri, Nov 20, 2026 at 07:22:11AM -0800, Raja Sai pranav wrote:
So pg_wal.old/000000010000000000000038 or
pg_wal/somedir/000000010000000000000038 is also taken as WAL. If such a
file comes first in the archive, the real pg_wal/ copy is then skipped as a
duplicate.
You are not mentioning which tool you are using is creating this
folder layer and particularly the reason behind this choice. In-core
configuration relies on max_wal_size as one way to control the size of
pg_wal/. So you are breaking at least one rule here.
--
Michael
On Sat, Nov 21, 2026 at 07:50:26AM +0900, Michael Paquier wrote:
You are not mentioning which tool you are using is creating this
folder layer and particularly the reason behind this choice.
No tool creates it. I made pg_wal.old/ by hand to test the check, and I
don't know of any tool or report where this happens in practice. I should
have said that in the first mail.
What I found is that member_is_wal_file() doesn't do what its comment says
("Skip files in subdirectories other than pg_wal/"). It compares only the
first six characters, so any member whose path starts with "pg_wal" is
taken as WAL. pg_basebackup copies whatever extra directories a user leaves
in the data directory, so such a directory ends up in base.tar. When that
happens, pg_verifybackup can report a backup as verified while its real WAL
is damaged.
If you think the case is too unlikely to need a test, I'm fine with keeping
only the code change, so that the check matches the comment, and dropping
the test.
Regards,
Sai Pranav
On Sat, Nov 21, 2026 04:20 AM, Michael Paquier <michael@paquier.xyz> wrote:
Show quoted text
On Fri, Nov 20, 2026 at 07:22:11AM -0800, Raja Sai pranav wrote:
So pg_wal.old/000000010000000000000038 or
pg_wal/somedir/000000010000000000000038 is also taken as WAL. If such a
file comes first in the archive, the real pg_wal/ copy is then skippedas a
duplicate.
You are not mentioning which tool you are using is creating this
folder layer and particularly the reason behind this choice. In-core
configuration relies on max_wal_size as one way to control the size of
pg_wal/. So you are breaking at least one rule here.
--
Michael