Compression of bigger WAL records

I

./pgbin/bin/pg_waldump

On Sun, 12 Jan 2025 at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

Hi hackers!

I propose a slight change to WAL compression: compress body of big records, if it's bigger than some threshold.

Hi,
initdb fails when configured with --without-zstd

```
reshke@ygp-jammy:~/postgres$ ./pgbin/bin/initdb -D db
The files belonging to this database system will be owned by user "reshke".
This user must also own the server process.

The database cluster will be initialized with locale "C.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

creating directory db ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "autovacuum_worker_slots" ... 16
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... 2025-01-12 18:10:47.657 UTC [4167965]
FATAL: zstd is not supported by this build
2025-01-12 18:10:47.657 UTC [4167965] PANIC: cannot abort transaction
1, it was already committed
Aborted (core dumped)
child process exited with exit code 134
initdb: removing data directory "db"
```

Also pg_waldump fails with

```
corrupted size vs. prev_size
Aborted (core dumped)
```

Best regards,
Kirill Reshke

Hi! Thanks for looking into this!

On 12 Jan 2025, at 23:36, Kirill Reshke <reshkekirill@gmail.com> wrote:

initdb fails when configured with --without-zstd

Yes, the patch is intended to demonstrate improvement when using Zstd.

On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

WAL_DEBUG and wal_compression are enabled for debugging purposes. Of course, I do not propose to turn them on by default.

And this does not work well --without-zstd.

Also pg_waldump fails with

```
corrupted size vs. prev_size
Aborted (core dumped)
```

I’ll fix that, thanks!
Also seems like I forgot to bump WAL_FILE_MAGIC…

What do you think about proposed approach?

Best regards, Andrey Borodin.

On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

I attach a prototype patch.

Here's v2, now it passes all the tests with wal_debug.

Some stats. On this test

create table a as select random() from generate_series(1,1e7);
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'lz4';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'pglz';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'zstd';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;

I observe WAL size of the index:
method HEAD patched
pglz 193 MB 193 MB
lz4 160 MB 132 MB
zstd 125 MB 97 MB

So, for lz4 and zstd this seems to be a significant reduction.

I'm planning to work on improving the patch quality.

Thanks!

Best regards, Andrey Borodin.

On Tue, 21 Jan 2025 at 23:24, "Andrey M. Borodin" <x4mmm@yandex-team.ru> wrote:

On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

I attach a prototype patch.

Here's v2, now it passes all the tests with wal_debug.

Some stats. On this test

create table a as select random() from generate_series(1,1e7);
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'lz4';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'pglz';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
set wal_compression to 'zstd';
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;

I observe WAL size of the index:
method HEAD patched
pglz 193 MB 193 MB
lz4 160 MB 132 MB
zstd 125 MB 97 MB

So, for lz4 and zstd this seems to be a significant reduction.

I'm planning to work on improving the patch quality.

Thanks!

Hi, Andrey Borodin

I find this feature interesting; however, it cannot be applied to the current
master (b35434b134b) due to commit 32a18cc0a73.

Applying: Compress big WAL records
.git/rebase-apply/patch:83: trailing whitespace.

.git/rebase-apply/patch:90: trailing whitespace.

.git/rebase-apply/patch:315: trailing whitespace.

.git/rebase-apply/patch:780: trailing whitespace.
else
error: contrib/pg_walinspect/pg_walinspect.c: does not match index
error: src/backend/access/rmgrdesc/xlogdesc.c: does not match index
error: src/backend/access/transam/xlog.c: does not match index
error: src/backend/access/transam/xloginsert.c: does not match index
error: src/backend/access/transam/xlogreader.c: does not match index
error: src/backend/utils/misc/guc_tables.c: does not match index
error: src/backend/utils/misc/postgresql.conf.sample: does not match index
error: src/include/access/xlog.h: does not match index
error: src/include/access/xloginsert.h: does not match index
error: src/include/access/xlogreader.h: does not match index
error: src/include/access/xlogrecord.h: does not match index
error: src/include/pg_config_manual.h: does not match index
error: src/test/recovery/t/026_overwrite_contrecord.pl: does not match index
error: patch failed: src/test/recovery/t/039_end_of_wal.pl:81
error: src/test/recovery/t/039_end_of_wal.pl: patch does not apply
Patch failed at 0001 Compress big WAL records
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

I see the patch compresses the WAL record according to the wal_compression,
IIRC the wal_compression is only used for FPI, right? Maybe we should update
the description of this parameter.

I see that the wal_compression_threshold defaults to 512. I wonder if you
chose this value based on testing or randomly.

--
Regrads,
Japin Li

On 2025/01/22 3:24, Andrey M. Borodin wrote:

On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:

I attach a prototype patch.

Here's v2, now it passes all the tests with wal_debug.

I like the idea of WAL compression more.

With the current approach, each backend needs to allocate memory twice
the size of the total WAL record. Right? One area is for the gathered
WAL record data (from rdt and registered_buffers), and the other is for
storing the compressed data. Could this lead to potential memory usage
concerns? Perhaps we should consider setting a limit on the maximum
memory each backend can use for WAL compression?

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On 23 Jan 2025, at 20:13, Japin Li <japinli@hotmail.com> wrote:

I find this feature interesting;

Thank you for your interest in the patch!

however, it cannot be applied to the current
master (b35434b134b) due to commit 32a18cc0a73.

PFA a rebased version.

I see the patch compresses the WAL record according to the wal_compression,
IIRC the wal_compression is only used for FPI, right? Maybe we should update
the description of this parameter.

Yes, I'll udpate documentation in future versions too.

I see that the wal_compression_threshold defaults to 512. I wonder if you
chose this value based on testing or randomly.

Voices in my head told me it's a good number.

On 28 Jan 2025, at 22:10, Fujii Masao <masao.fujii@oss.nttdata.com> wrote:

I like the idea of WAL compression more.

Thank you!

With the current approach, each backend needs to allocate memory twice
the size of the total WAL record. Right? One area is for the gathered
WAL record data (from rdt and registered_buffers), and the other is for
storing the compressed data.

Yes, exactly. And also a decompression buffer for each WAL reader.

Could this lead to potential memory usage
concerns? Perhaps we should consider setting a limit on the maximum
memory each backend can use for WAL compression?

Yes, the limit makes sense.

Also, we can reduce memory consumption by employing a streaming compression. Currently, I'm working on a prototype of such technology, because it would allow wholesale WAL compression. The idea is to reuse compression context from previous records to better compress new records. This would allow efficient compression of even very small records. However, there is exactly 0 chance to get it done in a decent shape before feature freeze.

The chances of getting currently proposed approach to v18 seems slim either... I'm hesitating to register this patch on the CF. What do you think?

Best regards, Andrey Borodin.

Hi Andery
I have a question ,If wal_compression_threshold is set to more than
the block size of the wal log, then the FPI is not compressed, and if so,
it might make sense to have a maximum value of this parameter that does not
exceed the block size of the wal log？

Best regards

On Thu, Jan 30, 2025 at 9:26 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

Show quoted text

On 23 Jan 2025, at 20:13, Japin Li <japinli@hotmail.com> wrote:

I find this feature interesting;

Thank you for your interest in the patch!

however, it cannot be applied to the current
master (b35434b134b) due to commit 32a18cc0a73.

PFA a rebased version.

I see the patch compresses the WAL record according to the

wal_compression,

IIRC the wal_compression is only used for FPI, right? Maybe we should

update

the description of this parameter.

Yes, I'll udpate documentation in future versions too.

I see that the wal_compression_threshold defaults to 512. I wonder if you
chose this value based on testing or randomly.

Voices in my head told me it's a good number.

On 28 Jan 2025, at 22:10, Fujii Masao <masao.fujii@oss.nttdata.com>

wrote:

I like the idea of WAL compression more.

Thank you!

With the current approach, each backend needs to allocate memory twice
the size of the total WAL record. Right? One area is for the gathered
WAL record data (from rdt and registered_buffers), and the other is for
storing the compressed data.

Yes, exactly. And also a decompression buffer for each WAL reader.

Could this lead to potential memory usage
concerns? Perhaps we should consider setting a limit on the maximum
memory each backend can use for WAL compression?

Yes, the limit makes sense.

Also, we can reduce memory consumption by employing a streaming
compression. Currently, I'm working on a prototype of such technology,
because it would allow wholesale WAL compression. The idea is to reuse
compression context from previous records to better compress new records.
This would allow efficient compression of even very small records. However,
there is exactly 0 chance to get it done in a decent shape before feature
freeze.

The chances of getting currently proposed approach to v18 seems slim
either... I'm hesitating to register this patch on the CF. What do you
think?

Best regards, Andrey Borodin.

On 31 Jan 2025, at 08:37, wenhui qiu <qiuwenhuifx@gmail.com> wrote:

Hi Andery
I have a question ,If wal_compression_threshold is set to more than the block size of the wal log, then the FPI is not compressed, and if so, it might make sense to have a maximum value of this parameter that does not exceed the block size of the wal log？

Oops, looks like I missed your question. Sorry for so long delay.

User might want to compress only megabyte+ records, there's nothing wrong with it. WAL record itself is capped by 1Gb (XLogRecordMaxSize), I do not see a reason to restrict wal_compression_threshold by lower value.

PFA rebased version.

Best regards, Andrey Borodin.

On 14 Jul 2025, at 23:22, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

PFA rebased version.

Here's a rebased version. Also I fixed a problem of possible wrong memory context used for allocating compression buffer.

Best regards, Andrey Borodin.

On Mon, Jan 12, 2026 at 2:54 AM Andrey Borodin <x4mmm@yandex-team.ru> wrote:

On 14 Jul 2025, at 23:22, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

PFA rebased version.

Here's a rebased version. Also I fixed a problem of possible wrong memory context used for allocating compression buffer.

Thanks for updating the patch!

With the v5 patch, I see the following compiler warning:

xlog.c:726:1: warning: unused function 'XLogGetRecordTotalLen'
[-Wunused-function]
726 | XLogGetRecordTotalLen(XLogRecord *record)
| ^~~~~~~~~~~~~~~~~~~~~

This seems to happen because XLogGetRecordTotalLen() is only used under
WAL_DEBUG. If that's correct, its definition should probably also be guarded
by WAL_DEBUG to avoid the warning.

cfbot reported a regression test failure with v5. Could you please
look into that?
https://cirrus-ci.com/build/5635306839343104

When I ran pg_waldump on WAL generated with wal_compression=pglz and
wal_compression_threshold=32, I got this error:

pg_waldump: error: error in WAL record at 0/02183BE0: could not
decompress record at 0/2183D10

Isn't this a bug?

+ XLogEnsureCompressionBuffer(MaxSizeOfXLogRecordBlockHeader + BLCKSZ);

XLogEnsureCompressionBuffer() is now called every time XLogRegisterBuffer(),
XLogRegisterBlock(), XLogRegisterData(), and XLogRegisterBufData() are invoked.
Why is that necessary? Wouldn't it be sufficient to
call XLogEnsureCompressionBuffer() once, with the total length,
just before XLogCompressRdt(rdt)?

v5 removes the ability to compress only full-page images, which is the current
wal_compression behavior. That may be disappointing for users who rely on
the existing semantics. Would it make more sense to keep the current behavior
and add a new feature to compress entire WAL records whose size exceeds
the specified threshold?

Regards,

--
Fujii Masao

Hi Fujii!

Thanks for the review, I'll address your feedback soon.

On 16 Jan 2026, at 20:44, Fujii Masao <masao.fujii@gmail.com> wrote:

Would it make more sense to keep the current behavior
and add a new feature to compress entire WAL records whose size exceeds
the specified threshold?

That's a very good idea! We don't need to replace current behavior, we can just complement it.
I'll implement this idea!

Best regards, Andrey Borodin.

On 16 Jan 2026, at 21:17, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

That's a very good idea! We don't need to replace current behavior, we can just complement it.
I'll implement this idea!

Here's the implementation. Previously existing buffers are now combined
into single allocation, which is GUC-controlled (you can add more memory).

However, now this buffer is just enough to accommodate most of records...
So, maybe we do not need a GUC at all, because keeping it minimal (same
consumption as before the patch) is just enough.

Now the patch essentially have no extra memory footprint, but allows to
save 25% of WAL on index creation (in case of random data).

User can force FPI-only compression by increasing wal_compression_threshold
to 1GB.

The decision chain is now a bit complicated:
- assemble record without compression FPIs
- try whole record compression
- if compression enlarged record fallback to FPI compression
I think the case can be simplified to "Try only one compression approach that
is expected to work, if not - insert uncompressed".

What do you think?

Best regards, Andrey Borodin.

Hello

+static void
+AllocCompressionBuffers(void)
+{
+ uint32 new_size = wal_compression_buffer;

This is called in the assign hook - and isn't that called before the
global variable is updated?

+ compressed_header->method = XLR_COMPRESS_LZ4;
+ compr_len = LZ4_compress_default((char *) &src_header[1], (char *)
&compressed_header[1],
+    orig_len, compressed_data_size);

compressed_header[1] has an offset of 32, but compressed_data_size
refers to the entire size, isn't there a possible buffer overrun here?
Same with ZSTD.

+/* Header prepended to a whole-record compressed WAL record */
+typedef struct XLogCompressionHeader
+{
+ XLogRecord record_header;
+ uint8 method; /* XLR_COMPRESS_* */
+ uint32 decompressed_length;
+} XLogCompressionHeader;

This has 3 bytes of uninitialized padding, is that okay? I remember
seeing a separate thread about possibly cleaning these up not long
ago.

+# Enable WAL compression for recovery tests.
+# lz4 is used here; 052_wal_compression.pl separately tests all methods.
+wal_compression = 'lz4'

Doesn't this test needs a check to require a build with compression flags?

+#wal_compression = lz4 # enables compression of
full-page writes;

But the default value is still off, isn't this example misleading?

+ total_len_decomp = -1; /* XLogCompressionHeader spans pages */

at multiple places, but this is an unsigned variable.

+  variable => 'wal_compression_buffer',
+  boot_val => '295972',
+  min => '295972',

Doesn't guc_params.dat support using macros instead of this hardcoded
magic number?

Hi, Andrey

On Mon, 09 Mar 2026 at 22:07, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

On 16 Jan 2026, at 21:17, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

That's a very good idea! We don't need to replace current behavior, we can just complement it.
I'll implement this idea!

Here's the implementation. Previously existing buffers are now combined
into single allocation, which is GUC-controlled (you can add more memory).

However, now this buffer is just enough to accommodate most of records...
So, maybe we do not need a GUC at all, because keeping it minimal (same
consumption as before the patch) is just enough.

Now the patch essentially have no extra memory footprint, but allows to
save 25% of WAL on index creation (in case of random data).

User can force FPI-only compression by increasing wal_compression_threshold
to 1GB.

The decision chain is now a bit complicated:
- assemble record without compression FPIs
- try whole record compression
- if compression enlarged record fallback to FPI compression
I think the case can be simplified to "Try only one compression approach that
is expected to work, if not - insert uncompressed".

What do you think?

Thanks for updating the patch. It seems a rebase is needed.

$ git am ~/v6-0001-Add-whole-record-WAL-compression-alongside-FPI-co.patch
Applying: Add whole-record WAL compression alongside FPI compression
error: patch failed: src/backend/access/transam/xloginsert.c:115
error: src/backend/access/transam/xloginsert.c: patch does not apply
Patch failed at 0001 Add whole-record WAL compression alongside FPI compression
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Best regards, Andrey Borodin.

[2. text/x-diff; v6-0001-Add-whole-record-WAL-compression-alongside-FPI-co.patch]...

--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.

Hi hackers!

This is v7, and it is now three patches. The thread had grown into one
patch doing several things at once, so I split it, and the first of the
three is useful on its own.

Sorry for the long silence, and for leaving several review comments
unanswered since March. I have tried to answer all of them below.

---- 0001: reuse a zstd compression context ----

This one is independent of the rest of the thread, so I am putting it
first as low-hanging fruit.

XLogCompressBackupBlock() calls ZSTD_compress(), which creates and
destroys a ZSTD_CCtx on every call. That context is not small: at the
default level ZSTD_estimateCCtxSize() reports 1.3MB. So a backend
running with wal_compression = zstd pays that allocation once per
full-page image, and a record can carry up to XLR_MAX_BLOCK_ID + 1 of
them.

The patch creates the context on first use and keeps it for the life of
the backend, compressing with ZSTD_compressCCtx(). On a statement that
logs 48k full-page images this is about 12% faster.

Pros: small, one file, no format change, and the WAL it produces is
identical byte for byte.

Cons: a backend that used zstd once holds the 1.3MB until it exits. I
think that is the right trade, but it is a real change in footprint for
a backend that compresses once and never again.

---- 0002: whole-record compression alongside FPI compression ----

This is the feature this thread started with, with Fujii-san's
suggestion applied: it complements per-FPI compression instead of
replacing it.

When a record is larger than wal_compression_threshold, it is compressed
as a single unit instead of compressing each full-page image on its own.
That wins whenever the images in one record share content, which is
typical of B-tree index builds. On CREATE INDEX over 10M random floats,
WAL drops from 160MB to 132MB with lz4 and from 125MB to 97MB with zstd.

Only lz4 and zstd take part now. pglz refuses input it cann't shrink
significantly, which a record made of full-page images rarely clears, so
in the earlier versions it cost a flatten and a compression attempt and
saved nothing measurable. It still compresses full-page images exactly as
before.

Pros: it reaches a redundancy that per-FPI compression cannot see, and
records below the threshold are untouched. Every record still decodes
on its own, so nothing about how WAL is read changes. Setting the
threshold above the largest possible record restores today's behaviour.

Cons:

1. Memory. The per-block compressed_page arrays are replaced by two
buffers carved out in one piece, one staging the record and one
taking the compressor output. Each is 274300 bytes with BLCKSZ 8192
and zstd compiled in, against 273372 bytes for the old arrays at
their 33-block maximum. So a backend with compression enabled holds
about twice what it held before. The staging buffer reproduces the
old allocation; the output buffer is genuinely new. It cannot be
removed without compressing straight off the rdt chain, and the
compressed length has to be known before WAL space is reserved, so
the output has to be materialized somewhere. Perhaps, I can just
assume the buffer of a twice smaller size, or somthing like that...

2. A low threshold can make WAL bigger. Whole-record compression
displaces per-FPI compression for the records it takes, and adds 8
bytes of header to each. On a pgbench run, threshold 32 produced 720
bytes of WAL per transaction against 622 at the default 512.

---- 0003: compress records against earlier records (WIP) ----

This is the direction I mentioned in the first message of this thread,
and it is not in a good shape yet. Just a prototype, it womewhat works but
the design discussion is needed.

Half the WAL a pgbench run produces is out of reach for per-record
compression. On a 1.6GB sample the median record was 72 bytes, and the
0.8% of records over 512 bytes held 47.5% of the bytes. A 72-byte
record does not compress on its own. It compresses very well against
the records before it.

Offline, on that sample, compressing each record on its own gives 1.76x.
Keeping a compressor across records, flushed after every record so the
record is complete when it is inserted, gives 2.93x across 8 streams.
The flush is not optional: WAL cannot leave a record's bytes inside a
compressor, and the compressed length has to be known before the space
is reserved.

The patch adds wal_compression_streams, a pool of streams a backend
leases while it compresses and inserts, so records enter a stream in the
order a reader will meet them. Currently a backend taking a lease over restarts
the stream, because the compressor state lives in the other backend and
shared memory cannot hold a libzstd object. I think I can fix it in future
iterations.

Once a record depends on the ones before it, some records have to stay
out of the scheme. The rule is not a resource-manager list but a
property: anything that someone reads by LSN without replaying what
precedes it has to stay readable on its own. Recovery finds the
checkpoint record from pg_control, and twophase.c reads a PREPARE record
from a stored LSN, so those are excluded, and XLOG_SWITCH with them.
This applies only with streams on; in 0002 every record is independent
and nothing is excluded.

What works: crash recovery, the data, amcheck, and reading a whole WAL
segment, because streams restart at segment boundaries.

What does not: a reader starting at an arbitrary LSN inside a segment
cannot decode, because it has no stream context for the records before
its start point. pg_waldump --start and pg_walinspect both fail that
way, and pg_rewind and logical decoding restart points would too. That
has to be solved before this is worth reviewing. I would welcome
opinions on the shape of the solution: my current thinking is that the
format has to let a reader find the nearest preceding stream restart.

It also costs throughput. On pgbench the stream lock, which is held
across compression and insertion, cost about 12% tps on my machine. I
have not measured that on a machine large enough for the WAL insert path
to be contended, and Andres has pointed out elsewhere that it already is
there.

---- Testing ----

Everything below is on a 16-core Linux box, built with AddressSanitizer
and --enable-cassert.

The workload walks the start offset of a compressed record across a
whole WAL page, emits messages sized on both sides of the threshold and
of the internal buffer bound, writes high-entropy payloads so the
compressor gives up, and takes four checkpoint-and-update rounds so
records carry full-page images. It then crashes the server and checks
that recovery reproduces two checksums, that pg_waldump and
pg_get_wal_records_info() decode the range, and that bt_index_check()
passes. wal_consistency_checking = all compares every replayed page
against the image in the record.

0001 and 0002 pass that with zero sanitizer reports, with compression
off, with zstd at the default and at threshold 32, with lz4, with pglz,
and with consistency checking on.

I also built all four combinations of --with-lz4 and --with-zstd with
-Werror, after Kirill's report above.

Two bugs in my own reader code turned up on the way, both now fixed in
0002. When a compressed record's header straddled a WAL page, the
reader reserved a decode slot sized for the compressed length and then
decoded the larger record into it. And the slot was not reset when the
read restarted, so a later pass could hand out a slot already in the
decode queue. Neither reproduces at the default threshold, which is why
they survived so long. I could not build a TAP test that fails before
the fix and passes after, so I have not added one.

---- Answers ----

On 12 Jan 2025, Kirill Reshke wrote:

initdb fails when configured with --without-zstd

Thank you, and sorry it took this long. Builds without the compression
libraries are handled now: the whole-record machinery is compiled out
when neither lz4 nor zstd is present, and the reader still recognizes
such a record and reports it properly rather than mis-decoding it. I
built all four combinations with -Werror.

On 23 Jan 2025, Japin Li wrote:

I see the patch compresses the WAL record according to the
wal_compression, IIRC the wal_compression is only used for FPI, right?
Maybe we should update the description of this parameter.

You are right, and I still owe this. postgresql.conf.sample is updated
in 0002, but the GUC description and the documentation are not. I will
do that before this soon.

I see that the wal_compression_threshold defaults to 512. I wonder if
you chose this value based on testing or randomly.

I answered this with a joke at the time, which it deserved less than a
real answer. I have measured it since: within per-record compression
the threshold barely matters, because small records do not compress on
their own. Compressing every record instead of only those over 512
bytes changed a 1.6GB sample by less than one percent. What the
threshold really controls is how much per-FPI compression gets
displaced, and lowering it can make WAL bigger, as in point 2 above.

For an outside data point, the usual recommendation for gzip over HTTP
is not to bother below roughly 860 bytes [1]https://webmasters.stackexchange.com/questions/31750/what-is-recommended-minimum-object-size-for-gzip-performance-benefits, on the same argument:
below that the header and the CPU cost outweigh what you save. That is
a different domain and WAL records are not HTTP responses, but the order
of magnitude agrees with what I measured. So 512 is a reasonable
default, for a better reason than the one I gave. If anything this
suggests it should be a little higher rather than lower, and I would not
object to 1kB if someone prefers a round number.

On 28 Jan 2025, Fujii Masao wrote:

With the current approach, each backend needs to allocate memory twice
the size of the total WAL record. [...] Perhaps we should consider
setting a limit on the maximum memory each backend can use for WAL
compression?

I tried that as a GUC and then removed it. The buffers are now a fixed
size, equal to the largest record that can be built, which is the same
amount the per-block arrays occupied at their maximum. A GUC did not
earn its place: the minimum useful value is already the value you want,
and it brought an assign-hook ordering bug with it, which Zsolt found.
The honest summary is point 1 above: it is 2x, not 1x, and I do not see
how to avoid the second buffer.

On 16 Jan 2026, Fujii Masao wrote:

With the v5 patch, I see the following compiler warning:
xloginsert.c:726:1: warning: unused function
'XLogGetRecordTotalLen'

Fixed since v6; it is guarded by WAL_DEBUG. Note my own tree builds
with -DWAL_DEBUG, so cfbot is what actually proves it.

cfbot reported a regression test failure with v5.

Should be gone. make check and the recovery suite pass, and the stress
matrix above passes under a sanitizer build.

When I ran pg_waldump on WAL generated with wal_compression=pglz and
wal_compression_threshold=32, I got this error:
pg_waldump: error: error in WAL record at 0/02183BE0: could not
decompress record at 0/2183D10
Isn't this a bug?

I could not reproduce it, and it is now moot for pglz specifically,
since pglz no longer takes part in whole-record compression. 0002 runs
pg_waldump over compressed WAL for every method as part of its test, and
the stress runs above do the same. If you still have the recipe I would
like to try it against v7.

XLogEnsureCompressionBuffer() is now called every time
XLogRegisterBuffer() [...] Why is that necessary?

It is not, and it is gone since v6. The buffers are allocated once,
outside any critical section.

v5 removes the ability to compress only full-page images [...] Would
it make more sense to keep the current behavior and add a new feature
to compress entire WAL records whose size exceeds the specified
threshold?

That was the right call and it is what 0002 does. Thank you.

On 9 Mar 2026, Zsolt Parragi wrote:

Thank you for reading it that closely. Every one of these was real.

+static void
+AllocCompressionBuffers(void)
+{
+ uint32 new_size = wal_compression_buffer;

This is called in the assign hook - and isn't that called before the
global variable is updated?

Yes. guc.c calls the assign hook and then assigns the variable, so the
hook saw the old value, and raising the GUC did not grow the buffers.
XLogCompressRdt() would then flatten a larger record into a smaller
buffer, which is a heap overflow in a non-assert build. The GUC is gone
now, and the remaining hook reads newval.

compressed_header[1] has an offset of 32, but compressed_data_size
refers to the entire size, isn't there a possible buffer overrun here?
Same with ZSTD.

Correct. The destination now gets the space that is really left, after
the header.

This has 3 bytes of uninitialized padding, is that okay?

It was not. The header is zeroed before it is filled, so the bytes that
reach disk are deterministic. I also reordered the fields, and
SizeOfXLogCompressedRecord is now plain sizeof(), which keeps it in
agreement with the pointer the payload is written to.

Doesn't this test needs a check to require a build with compression
flags?

Fixed by using pglz for the suite-wide config, since it is always
available. 052 covers lz4 and zstd separately and skips what a build
lacks.

But the default value is still off, isn't this example misleading?

That was a leftover from my testing. postgresql.conf.sample says off
again.

+ total_len_decomp = -1; /* XLogCompressionHeader spans pages */
at multiple places, but this is an unsigned variable.

Replaced with an explicit boolean. While fixing it I found that the
branch it belonged to was wrong in a worse way: it reserved a decode
slot sized for the compressed length and then decoded the decompressed
record into it.

Doesn't guc_params.dat support using macros instead of this hardcoded
magic number?

It does, but the GUC is gone so the number with it.

On 10 Mar 2026, Japin Li wrote:

Thanks for updating the patch. It seems a rebase is needed.

Done, on top of master as of this week. Sorry for the wait.

Thank you!

Best regards, Andrey Borodin.

[0]: /messages/by-id/4DC38068-976E-4A84-8EE6-4EFACBBD927A@yandex-team.ru
[1]: https://webmasters.stackexchange.com/questions/31750/what-is-recommended-minimum-object-size-for-gzip-performance-benefits

On 26 Jul 2026, at 21:09, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

This is v7

Hi hackers!

This is v8. Still three patches. 0003 now answers the question I left
open in v7 - a reader can start at an arbitrary LSN. 0001 grew to
cover decompression as well, which turned out to be the bigger win, and
running the test suites with the feature actually turned on found a
number of bugs.

Numbers are from two machines, 4-core for 0002 and the compression side
of 0001, 16-core for 0003 and the redo figures, both built with -O2 and
without assertions, fsync off, shared_buffers 8GB. Each was repeated,
where a number moved between runs I say so.

---- 0001: reuse zstd contexts, both directions ----

Still the piece that is useful on its own, and it grew since v7: it now
keeps the decompression context too, which turns out to matter more.

XLogCompressBackupBlock() calls ZSTD_compress() and RestoreBlockImage()
calls ZSTD_decompress(). Both create and destroy a context per call. At
the default level ZSTD_estimateCCtxSize() reports 1.3MB for the
contex. So zstd pays an allocation per full-page image on the
way in, and one per image again on the way out. The patch creates each
on first use and keeps it, the compressor for the life of the backend,
the decompressor in XLogReaderState.

The reading side is where this shows, because one startup process
replays every image. Redoing 200MB of page images:

zstd, master 4.24s zstd, patched 2.33s
lz4, master 1.59s lz4, patched 1.59s

A sequential scan that sets hint bits on a freshly checkpointed table
with wal_log_hints on runs about 26% faster.

This also bears on the "WAL compression setting after PostgreSQL LZ4
default change" thread, where the order zstd -> lz4 -> pglz is proposed
for what "on" should mean. On master, replaying zstd-compressed images
costs 2.7x what lz4 costs, even though zstd wrote 38% less WAL; with the
context kept, that gap falls to 1.5x.

Cons: a backend that used zstd once holds the 1.3MB until it exits, and
a reader holds a decompression context, which is far smaller.

---- 0002: whole-record compression alongside FPI compression ----

Unchanged in design from v7, plus the documentation that was missing.

When a record is larger than wal_compression_threshold it is compressed
as a single unit rather than each full-page image separately, which wins
whenever the images in one record share content.

On CREATE INDEX over 10M random doubles, with the generator seeded so
the runs are comparable:

zstd 144.8MB -> 116.9MB (-19%)
lz4 189.5MB -> 161.0MB (-15%)

Pros: it reaches redundancy that per-FPI compression cannot see, records
below the threshold are untouched, and every record still decodes on its
own, so nothing about how WAL is read changes. Setting the threshold
above the largest possible record restores today's behaviour.

Cons, both unchanged from v7:

1. Memory. A backend with compression enabled holds two 274300-byte
buffers where it used to hold one array of about the same total size:
the staging buffer reproduces the old allocation, the output buffer
is new. The compressed length has to be known before WAL space is
reserved, so the output has to be materialized somewhere.

2. A low threshold can make WAL bigger, because whole-record compression
displaces per-FPI compression for the records it takes and adds
header to each.

---- 0003: compress records against earlier records (WIP) ----

Still WIP, but no longer blocked on the reader problem.

The motivation is unchanged: half the WAL a pgbench run produces is out
of reach for per-record compression, because the median record is far
too small to compress on its own but compresses well against the records
before it.

What is new in v8 is how a reader starts in the middle of WAL.

Every stream starts over at fixed 4MB boundaries. The writer enforces
that rather than hoping for it: a record that would continue a stream
past the next boundary is refused its reserved position and built again
against a stream that starts over. The check is one comparison against
CurrBytePos under the spinlock that reservation already holds; the
boundary is converted to a byte position outside the lock. It fires
about once per stream per 4MB, and I could not measure its cost.

A reader that wants to start at some LSN then rewinds to the boundary
below it and reads forward, which rebuilds the decompressors, and stops
short of the record it was asked for: feeding a record to its
decompressor twice would leave it in a state its successors were not
compressed against. That is XLogBeginReadStreamed(), and pg_waldump,
pg_walinspect, logical decoding, walsummarizer and pg_rewind all use it.
A record whose stream has not been seen to start over refuses to
decompress rather than decoding whatever the bytes happen to mean.

Why a fixed distance and not the WAL segment: how far a reader rewinds
should not change when a cluster is initialised with a different segment
size, and 64MB segments are not unusual. 4MB divides both 16MB and 64MB
segments, so a boundary is also always a page start. Compression is
insensitive to the value - pgbench emits the same WAL per transaction
to within 2% anywhere between 1MB and 64MB - so it is chosen for the
readers.

Numbers below are from a 16-core machine, pgbench scale 100, fsync off
so that this measures the feature and not the disk. WAL bytes per
transaction and throughput, with the stream count matched to the client
count:

clients streams off streams = clients
1 1720 / 2456 1086 / 2340 -37% WAL, -5% tps
8 701 / 18432 530 / 17326 -24% WAL, -6% tps
32 544 / 43729 433 / 44111 -20% WAL, no cost
64 542 / 46173 448 / 39499 -17% WAL, -12% tps

The 32- and 64-client rows answer the question I could not answer in v7:
the stream lease, which is held across compression and insertion, does
not show up as contention when the insert path is already busy. At 32
clients with 32 streams there is no measurable cost at all.

What does show up is that the stream count has to track concurrency.
Eight streams buy 4% at 32 clients and nothing at 64:

64 clients, streams 0 / 8 / 64 -> 542 / 534 / 448 bytes per txn

Building the third patch with wal_compression_streams = 0 reproduces the
second patch's numbers, so the cost is in using the feature, not in
carrying it.

Two other shapes, WAL volume only:

wide UPDATE of 500k rows 223.9MB -> 154.6MB (-31%)
COPY of 3M rows 65.4MB -> 68.6MB (+5%)

The COPY case is the honest counterexample: those records are large and
already compress well on their own, so the stream adds header and buys
nothing. I do not think that argues against the feature, but it does
argue that turning it on should stay a choice.

Memory, peak RSS with 64 clients writing and then a full pg_waldump over
what they wrote:

streams backend peak pg_waldump peak
0 146.6 MB 3.4 MB
8 150.5 MB 9.0 MB
64 150.5 MB 51.5 MB

The writing side costs about 4MB per backend and does not grow with the
stream count, because a backend keeps one compressor rather than one per
stream. The reading side costs about 0.75MB per stream, and every
reader pays it: the startup process, a walsender doing logical decoding,
pg_waldump.

Costs, as I see them:

1. Throughput, when streams are pushed as high as the client count: 12%
at 64 clients on 16 cores. At and below one stream per core I could
not measure a cost.

2. Memory. 0.75MB per stream for every reader, as above.

3. WAL retention. A replication slot has to keep the WAL back to the
reset boundary below what it needs itself, so up to 4MB more.

The same rewind costs reading, not just retention: a reader that
starts in the middle re-reads up to 4MB to rebuild the decompressors,
and it does so whether or not the WAL it is about to read holds any
streams at all. For readers that start once that is nothing; for
walsummarizer, which starts afresh per summary file, it is up to 4MB
per 16MB summarized. Making the rewind happen only when a record
actually turns out to need it is the obvious answer and I have not
got it working yet.

4. Records that someone reads by LSN without replaying what precedes
them have to stay out of the scheme: the checkpoint records including
XLOG_CHECKPOINT_REDO, XLOG_END_OF_RECOVERY, XLOG_SWITCH and PREPARE.

I could not measure the fsync=on case usefully. The disk I have caps at
82MB/s and repeats of one configuration differed by a factor of two, so
I have no throughput claim there; WAL volume did reproduce, 15-25%
lower with streams, in line with the numbers above.

What I would still like opinions on: whether refusing a reserved
position is an acceptable thing for an insertion path to do, and how to
resolve the tension the numbers above show: the ratio wants roughly one
stream per writing backend, while each stream costs every reader
0.75MB.
wal_compression_streams currently caps at 64, which is already generous
for a reader and far short of the backend count on a busy server.

WDYT?

Best regards, Andrey Borodin.

Hi, Andrey

Thanks for updating the patches.

On Fri, 07 Aug 2026 at 18:15, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

On 26 Jul 2026, at 21:09, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

This is v7

Hi hackers!

This is v8. Still three patches. 0003 now answers the question I left
open in v7 - a reader can start at an arbitrary LSN. 0001 grew to
cover decompression as well, which turned out to be the bigger win, and
running the test suites with the feature actually turned on found a
number of bugs.

Numbers are from two machines, 4-core for 0002 and the compression side
of 0001, 16-core for 0003 and the redo figures, both built with -O2 and
without assertions, fsync off, shared_buffers 8GB. Each was repeated,
where a number moved between runs I say so.

---- 0001: reuse zstd contexts, both directions ----

Still the piece that is useful on its own, and it grew since v7: it now
keeps the decompression context too, which turns out to matter more.

XLogCompressBackupBlock() calls ZSTD_compress() and RestoreBlockImage()
calls ZSTD_decompress(). Both create and destroy a context per call. At
the default level ZSTD_estimateCCtxSize() reports 1.3MB for the
contex. So zstd pays an allocation per full-page image on the
way in, and one per image again on the way out. The patch creates each
on first use and keeps it, the compressor for the life of the backend,
the decompressor in XLogReaderState.

The reading side is where this shows, because one startup process
replays every image. Redoing 200MB of page images:

zstd, master 4.24s zstd, patched 2.33s
lz4, master 1.59s lz4, patched 1.59s

A sequential scan that sets hint bits on a freshly checkpointed table
with wal_log_hints on runs about 26% faster.

This also bears on the "WAL compression setting after PostgreSQL LZ4
default change" thread, where the order zstd -> lz4 -> pglz is proposed
for what "on" should mean. On master, replaying zstd-compressed images
costs 2.7x what lz4 costs, even though zstd wrote 38% less WAL; with the
context kept, that gap falls to 1.5x.

Cons: a backend that used zstd once holds the 1.3MB until it exits, and
a reader holds a decompression context, which is far smaller.

---- 0002: whole-record compression alongside FPI compression ----

Unchanged in design from v7, plus the documentation that was missing.

When a record is larger than wal_compression_threshold it is compressed
as a single unit rather than each full-page image separately, which wins
whenever the images in one record share content.

On CREATE INDEX over 10M random doubles, with the generator seeded so
the runs are comparable:

zstd 144.8MB -> 116.9MB (-19%)
lz4 189.5MB -> 161.0MB (-15%)

Pros: it reaches redundancy that per-FPI compression cannot see, records
below the threshold are untouched, and every record still decodes on its
own, so nothing about how WAL is read changes. Setting the threshold
above the largest possible record restores today's behaviour.

Cons, both unchanged from v7:

1. Memory. A backend with compression enabled holds two 274300-byte
buffers where it used to hold one array of about the same total size:
the staging buffer reproduces the old allocation, the output buffer
is new. The compressed length has to be known before WAL space is
reserved, so the output has to be materialized somewhere.

2. A low threshold can make WAL bigger, because whole-record compression
displaces per-FPI compression for the records it takes and adds
header to each.

---- 0003: compress records against earlier records (WIP) ----

Still WIP, but no longer blocked on the reader problem.

The motivation is unchanged: half the WAL a pgbench run produces is out
of reach for per-record compression, because the median record is far
too small to compress on its own but compresses well against the records
before it.

What is new in v8 is how a reader starts in the middle of WAL.

Every stream starts over at fixed 4MB boundaries. The writer enforces
that rather than hoping for it: a record that would continue a stream
past the next boundary is refused its reserved position and built again
against a stream that starts over. The check is one comparison against
CurrBytePos under the spinlock that reservation already holds; the
boundary is converted to a byte position outside the lock. It fires
about once per stream per 4MB, and I could not measure its cost.

A reader that wants to start at some LSN then rewinds to the boundary
below it and reads forward, which rebuilds the decompressors, and stops
short of the record it was asked for: feeding a record to its
decompressor twice would leave it in a state its successors were not
compressed against. That is XLogBeginReadStreamed(), and pg_waldump,
pg_walinspect, logical decoding, walsummarizer and pg_rewind all use it.
A record whose stream has not been seen to start over refuses to
decompress rather than decoding whatever the bytes happen to mean.

Why a fixed distance and not the WAL segment: how far a reader rewinds
should not change when a cluster is initialised with a different segment
size, and 64MB segments are not unusual. 4MB divides both 16MB and 64MB
segments, so a boundary is also always a page start. Compression is
insensitive to the value - pgbench emits the same WAL per transaction
to within 2% anywhere between 1MB and 64MB - so it is chosen for the
readers.

Numbers below are from a 16-core machine, pgbench scale 100, fsync off
so that this measures the feature and not the disk. WAL bytes per
transaction and throughput, with the stream count matched to the client
count:

clients streams off streams = clients
1 1720 / 2456 1086 / 2340 -37% WAL, -5% tps
8 701 / 18432 530 / 17326 -24% WAL, -6% tps
32 544 / 43729 433 / 44111 -20% WAL, no cost
64 542 / 46173 448 / 39499 -17% WAL, -12% tps

The 32- and 64-client rows answer the question I could not answer in v7:
the stream lease, which is held across compression and insertion, does
not show up as contention when the insert path is already busy. At 32
clients with 32 streams there is no measurable cost at all.

What does show up is that the stream count has to track concurrency.
Eight streams buy 4% at 32 clients and nothing at 64:

64 clients, streams 0 / 8 / 64 -> 542 / 534 / 448 bytes per txn

Building the third patch with wal_compression_streams = 0 reproduces the
second patch's numbers, so the cost is in using the feature, not in
carrying it.

Two other shapes, WAL volume only:

wide UPDATE of 500k rows 223.9MB -> 154.6MB (-31%)
COPY of 3M rows 65.4MB -> 68.6MB (+5%)

The COPY case is the honest counterexample: those records are large and
already compress well on their own, so the stream adds header and buys
nothing. I do not think that argues against the feature, but it does
argue that turning it on should stay a choice.

Memory, peak RSS with 64 clients writing and then a full pg_waldump over
what they wrote:

streams backend peak pg_waldump peak
0 146.6 MB 3.4 MB
8 150.5 MB 9.0 MB
64 150.5 MB 51.5 MB

The writing side costs about 4MB per backend and does not grow with the
stream count, because a backend keeps one compressor rather than one per
stream. The reading side costs about 0.75MB per stream, and every
reader pays it: the startup process, a walsender doing logical decoding,
pg_waldump.

Costs, as I see them:

1. Throughput, when streams are pushed as high as the client count: 12%
at 64 clients on 16 cores. At and below one stream per core I could
not measure a cost.

2. Memory. 0.75MB per stream for every reader, as above.

3. WAL retention. A replication slot has to keep the WAL back to the
reset boundary below what it needs itself, so up to 4MB more.

The same rewind costs reading, not just retention: a reader that
starts in the middle re-reads up to 4MB to rebuild the decompressors,
and it does so whether or not the WAL it is about to read holds any
streams at all. For readers that start once that is nothing; for
walsummarizer, which starts afresh per summary file, it is up to 4MB
per 16MB summarized. Making the rewind happen only when a record
actually turns out to need it is the obvious answer and I have not
got it working yet.

4. Records that someone reads by LSN without replaying what precedes
them have to stay out of the scheme: the checkpoint records including
XLOG_CHECKPOINT_REDO, XLOG_END_OF_RECOVERY, XLOG_SWITCH and PREPARE.

I could not measure the fsync=on case usefully. The disk I have caps at
82MB/s and repeats of one configuration differed by a factor of two, so
I have no throughput claim there; WAL volume did reproduce, 15-25%
lower with streams, in line with the numbers above.

What I would still like opinions on: whether refusing a reserved
position is an acceptable thing for an insertion path to do, and how to
resolve the tension the numbers above show: the ratio wants roughly one
stream per writing backend, while each stream costs every reader
0.75MB.
wal_compression_streams currently caps at 64, which is already generous
for a reader and far short of the backend count on a busy server.

WDYT?

The patches cannot be applied to the current tree because they conflict with
commits 931c9701f25, 18992dc9d98, and b614de4876b.

Below are some initial review comments.

v8-0002

1.
+			report_invalid_record(state,
+								  "could not decompress record at %X/%08X compressed with %s not supported by build",
+								  LSN_FORMAT_ARGS((XLogRecPtr) recptr), "lz4");
+			return NULL;

The casting of XLogRecPtr is unnecessary.

2.
+			report_invalid_record(state,
+								  "could not decompress record at %X/%08X compressed with %s not supported by build",
+								  LSN_FORMAT_ARGS((XLogRecPtr) recptr), "zstd");
+			return NULL;

Same as above.

3.
+			report_invalid_record(state,
+								  "could not decompress record at %X/%08X compressed with unknown method",
+								  LSN_FORMAT_ARGS((XLogRecPtr) recptr));
+			return NULL;

Same as above.

v8-0003

1.
+				state->stream_dctx = palloc0(sizeof(void *) * XLR_MAX_STREAMS);
+				state->stream_ready = palloc0(sizeof(bool) * XLR_MAX_STREAMS);

I'd prefer use the palloc0_array() macro.

Best regards, Andrey Borodin.

--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.

HI Japin

+ state->stream_dctx =

palloc0(sizeof(void *) * XLR_MAX_STREAMS);

+ state->stream_ready =

palloc0(sizeof(bool) * XLR_MAX_STREAMS);

I'd prefer use the palloc0_array() macro.

state->stream_dctx = palloc0_array(void *, XLR_MAX_STREAMS);
state->stream_ready = palloc0_array(bool, XLR_MAX_STREAMS);
Agree +1

Thanks

On 14 Aug 2026, at 08:32, Japin Li <japinli@hotmail.com> wrote:

The patches cannot be applied to the current tree because they conflict with
commits 931c9701f25, 18992dc9d98, and b614de4876b.

Below are some initial review comments.

Hi Japin,

Thanks for the review, and for catching that the patches had gone stale.

Rebased, v9 is attached. Three of those conflicts were worth more than
a mechanical fixup.

931c9701f25 changed what wal_compression = on means, so the sample file
and the docs now describe "the first of zstd, lz4, pglz that is
available" and my paragraph about whole-record compression sits under
that rather than replacing it. I also removed the sentence saying that
"on" is a historical spelling of pglz, which that commit made untrue.

That commit is also why 0001 matters more than it did. With "on" now
reaching for zstd first, every installation that turns compression on
without naming an algorithm gets the codec that allocates a context per
full-page image, in both directions.

18992dc9d98 added a check that a record's length does not exceed
XLogRecordMaxSize before reconstructing it. 0002 splits that length in
two, the bytes a record occupies in WAL and the bytes it decodes to, so
the check needed a side. It now guards the physical length, which is
what the reassembly buffer actually holds. I also tightened the
decompressed-length sanity check from MaxAllocSize to XLogRecordMaxSize,
since that is the limit XLogRecordAssemble() enforces on the writing
side, and it seemed wrong to accept on read what cannot be written. A
plain textual rebase left that one compiling against a variable that no
longer exists, so thank you for the nudge to look.

b614de4876b took the test number 055, so the test in the series is now
056_wal_compression.pl.

The casting of XLogRecPtr is unnecessary.

Fixed, all three.

I'd prefer use the palloc0_array() macro.

Done, for both stream_dctx and stream_ready.

The Windows CI run on the first rebase also turned up a bug of my own,
now fixed here: XLOGShmemAttach() re-establishes WALInsertLocks for a
process that attaches to shared memory rather than inheriting it, and
0003 added a second such pointer without doing the same for it. Under
EXEC_BACKEND the startup process therefore found a null slot array and
died on the first record it tried to insert. The slot pointer now lives
in XLogCtl->Insert beside WALInsertLocks and is restored on attach.
I reproduced that on Linux by building with -DEXEC_BACKEND rather than
guessing from the Windows backtrace. My Windows machine went offline
while I'm on vacation overseas... But I think everything should work
now, even on Windows.

Best regards, Andrey Borodin.