Use WALReadFromBuffers in more places

Hi, Bharath. I've been testing this. It's cool. Is there any way we could
monitor the hit rate about directly reading from WAL buffers by exporting
to some views?

---

Regards, Jingtang

On Wed, May 8, 2024 at 9:51 AM Jingtang Zhang <mrdrivingduck@gmail.com> wrote:

Hi, Bharath. I've been testing this. It's cool. Is there any way we could
monitor the hit rate about directly reading from WAL buffers by exporting
to some views?

Thanks for looking into this. I used purpose-built patches for
verifying the WAL buffers hit ratio, please check
USE-ON-HEAD-Collect-WAL-read-from-file-stats.txt and
USE-ON-PATCH-Collect-WAL-read-from-buffers-and-file-stats.txt at
/messages/by-id/CALj2ACU9cfAcfVsGwUqXMace_7rfSBJ7+hXVJfVV1jnspTDGHQ@mail.gmail.com.
In the long run, it's better to extend what's proposed in the thread
/messages/by-id/CALj2ACU_f5_c8F+xyNR4HURjG=Jziiz07wCpQc=AqAJUFh7+8w@mail.gmail.com.
I haven't had a chance to dive deep into that thread yet, but a quick
glance over v8 patch tells me that it hasn't yet implemented the idea
of collecting WAL read stats for both walsenders and the backends
reading WAL. If that's done, we can extend it for WAL read from WAL
buffers.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

Hi Bharath,

I spent some time examining the patch. Here are my observations from the review.

- I believe there’s no need for an extra variable ‘nbytes’ in this
context. We can repurpose the ‘count’ variable for the same function.
If necessary, we might think about renaming ‘count’ to ‘nbytes’.

- The operations performed by logical_read_xlog_page() and
read_local_xlog_page_guts() are identical. It might be beneficial to
create a shared function to minimize code repetition.

Best Regards,
Nitin Jadhav
Azure Database for PostgreSQL
Microsoft

On Mon, May 13, 2024 at 12:17 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

Hi,

On Sat, Jun 8, 2024 at 5:24 PM Nitin Jadhav <nitinjadhavpostgres@gmail.com>
wrote:

I spent some time examining the patch. Here are my observations from the

review.

Thanks.

- I believe there’s no need for an extra variable ‘nbytes’ in this
context. We can repurpose the ‘count’ variable for the same function.
If necessary, we might think about renaming ‘count’ to ‘nbytes’.

'count' variable can't be altered once determined as the page_read
callbacks need to return the total number of bytes read. However, I ended
up removing 'nbytes' like in the attached v2 patch.

- The operations performed by logical_read_xlog_page() and
read_local_xlog_page_guts() are identical. It might be beneficial to
create a shared function to minimize code repetition.

IMO, creating another function to just wrap two other functions doesn't
seem good to me.

I attached v2 patches for further review. No changes in 0002 patch.

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

Hi all.

I've been back to this patch for a while recently. I witness that if a WAL
writer works fast, the already flushed WAL buffers will be zeroed out and
re-initialized for future use by AdvanceXLInsertBuffer in
XLogBackgroundFlush, so that WALReadFromBuffers will miss even though the
space of WAL buffer is enough. It is much more unfriendly for logical
walsenders than physical walsenders, because logical ones consume WAL
slower than physical ones due to the extra decoding phase. Seems that the
aim
of AdvanceXLInsertBuffer in WAL writer contradicts with our reading from
WAL buffer. Any thoughts?

Hi,

On Tue, Oct 15, 2024 at 1:22 AM Jingtang Zhang <mrdrivingduck@gmail.com> wrote:

I've been back to this patch for a while recently. I witness that if a WAL
writer works fast, the already flushed WAL buffers will be zeroed out and
re-initialized for future use by AdvanceXLInsertBuffer in
XLogBackgroundFlush, so that WALReadFromBuffers will miss even though the
space of WAL buffer is enough. It is much more unfriendly for logical
walsenders than physical walsenders, because logical ones consume WAL
slower than physical ones due to the extra decoding phase. Seems that the aim
of AdvanceXLInsertBuffer in WAL writer contradicts with our reading from
WAL buffer. Any thoughts?

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files. The no-locks design ensures it
never gets in the way of backends generating WAL, which is critical
for overall system performance.

I rebased and attached the v3 patch. I discarded the test extension
patch that demonstrated WALReadFromBuffers' behavior (i.e., waiting
for WAL to be fully copied to WAL buffers with
WaitXLogInsertionsToFinish), as I believe the comment at the top of
WALReadFromBuffers is sufficient documentation. I can reintroduce the
test extension if there's interest.

Thoughts?

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

Hi Bharath,

Hi,

Commit 91f2cae7a4e that introduced WALReadFromBuffers only used it for
physical walsenders. It can also be used in more places benefitting
logical walsenders, backends running pg_walinspect and logical
decoding functions if the WAL is available in WAL buffers. I'm
attaching a 0001 patch for this.

Thank you for working on this. It seems like a useful optimization. Do you
have any information
on how much this improves the performance of the new callers of
WALReadFromBuffers?

Regarding the v3 version of the patch, do you also intend to include other
callers of WALRead,
such as walsummarizer.c and pg_waldump.c?

In WALReadFromBuffers, the buffer scan currently stops when it encounters a
buffer that
doesn't have the needed WAL page. However, it's possible to continue
scanning past the
missing page and find other relevant pages further along. By keeping track
of which pages
are missing, we could read only those specific pages from files, instead of
reading everything
after the first missing page. I am wondering if this was taken into account
during the design
of the function.

Thank you,
Rahila Syed

Hi~

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files. The no-locks design ensures it
never gets in the way of backends generating WAL, which is critical
for overall system performance.

Yes, it is actually an interesting thing, beyond current topic. Since
we are using buffered I/O, even though we cannot read from WAL buffer due
to the opportunistic AdvanceXLInsertBuffer by WAL writer, later WALRead may
still find the page inside OS page cache, with high probability, because the
page has just been written out. So WALRead will be fast, too.

But if we are moving forward to direct I/O some day in the future, the cost
of WALReadFromBuffers and WALRead might be obvious. Maybe the opportunistic
WAL buffer initialization could keep a small ratio of old pages inside WAL
buffer so these pages can still be hit by WALReadFromBuffers.

I rebased and attached the v3 patch.

The v3 patch LGTM.

—
Regards,
Jingtang

Alibaba Cloud

On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrote:

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files.

IIRC, one motivation (perhaps the primary motivation?) was to make it
possible to read buffers before they are flushed. It was always
possible to read already-flushed buffers.

The benefit of reading unflushed buffers is that we can replicate the
WAL sooner (though it can't be replayed until the primary flushes it).
Is that right?

Regards,
Jeff Davis

Hi,

On Mon, Sep 22, 2025 at 8:56 PM Jeff Davis <pgsql@j-davis.com> wrote:

On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrote:

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files.

IIRC, one motivation (perhaps the primary motivation?) was to make it
possible to read buffers before they are flushed. It was always
possible to read already-flushed buffers.

The benefit of reading unflushed buffers is that we can replicate the

WAL sooner (though it can't be replayed until the primary flushes it).
Is that right?

I'm not certain about the primary motivation, but as it stands,
WALReadFromBuffers only reads WAL records present in buffers up to the
flush pointer. This is because XLogSendPhysical currently sends records
only up to the flush pointer, not beyond.

I am currently testing a patch developed by Melih Mutlu that implements the
functionality you described, sending unflushed buffers during physical
replication. After some tuning, the patch has shown a 5 percent improvement
in TPS for synchronous replication with remote_write. I am working on
further improving the patch before sharing it on the hackers mailing list.

Thank you,
Rahila Syed

Hi,

On Mon, Sep 22, 2025 at 8:26 AM Jeff Davis <pgsql@j-davis.com> wrote:

On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrote:

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files.

IIRC, one motivation (perhaps the primary motivation?) was to make it
possible to read buffers before they are flushed. It was always
possible to read already-flushed buffers.

The benefit of reading unflushed buffers is that we can replicate the
WAL sooner (though it can't be replayed until the primary flushes it).
Is that right?

Right. Reading unflushed WAL buffers for replication was one of the
motivations. But, in general, WALReadFromBuffers has more benefits
since it lets WAL buffers act as a cache for reads, avoiding the need
to re-read WAL from disk for (both physical and logical) replication.
For example, it makes the use of direct I/O for WAL more realistic and
can provide significant performance benefits [1]/messages/by-id/20230114203403.4zpg72fw2qb34awf@awork3.anarazel.de.

[1]: /messages/by-id/20230114203403.4zpg72fw2qb34awf@awork3.anarazel.de

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Wed, 2025-09-24 at 07:26 -0700, Bharath Rupireddy wrote:

Right. Reading unflushed WAL buffers for replication was one of the
motivations. But, in general, WALReadFromBuffers has more benefits
since it lets WAL buffers act as a cache for reads, avoiding the need
to re-read WAL from disk for (both physical and logical) replication.
For example, it makes the use of direct I/O for WAL more realistic
and
can provide significant performance benefits [1].

Is it possible to do a POC that shows the potential benefit, or are we
still too far away?

Regards,
Jeff Davis

Hi,

On Wed, Sep 24, 2025 at 11:27 AM Jeff Davis <pgsql@j-davis.com> wrote:

On Wed, 2025-09-24 at 07:26 -0700, Bharath Rupireddy wrote:

Right. Reading unflushed WAL buffers for replication was one of the
motivations. But, in general, WALReadFromBuffers has more benefits
since it lets WAL buffers act as a cache for reads, avoiding the need
to re-read WAL from disk for (both physical and logical) replication.
For example, it makes the use of direct I/O for WAL more realistic
and
can provide significant performance benefits [1].

Is it possible to do a POC that shows the potential benefit, or are we
still too far away?

Thanks for looking into this. I did performance analysis with WAL directo
I/O to see how reading from WAL buffers affects walsenders:
/messages/by-id/CALj2ACV6rS+7iZx5+oAvyXJaN4AG-djAQeM1mrM=YSDkVrUs7g@mail.gmail.com.
Following is from that thread. Please let me know if you have any specific
cases in mind. I'm happy to run the same test for logical replication.

It helps WAL DIO; since there's no OS
page cache, using WAL buffers as read cache helps a lot. It is clearly
evident from my experiment with WAL DIO patch [1], see the results [2]Test case is an insert pgbench workload. clients HEAD | WAL DIO | WAL DIO & WAL BUFFERS READ | WAL BUFFERS READ 1 1404 1070 1424 1375 2 1487 796 1454 1517 4 3064 1743 3011 3019 8 6114 3556 6026 5954 16 11560 7051 12216 12132 32 23181 13079 23449 23561 64 43607 26983 43997 45636 128 80723 45169 81515 81911 256 110925 90185 107332 114046 512 119354 109817 110287 117506 768 112435 105795 106853 111605 1024 107554 105541 105942 109370 2048 88552 79024 80699 90555 4096 61323 54814 58704 61743
and attached graph. As expected, WAL DIO brings down the TPS, whereas
WAL buffers read i.e. this patch brings it up.

[2]: Test case is an insert pgbench workload. clients HEAD | WAL DIO | WAL DIO & WAL BUFFERS READ | WAL BUFFERS READ 1 1404 1070 1424 1375 2 1487 796 1454 1517 4 3064 1743 3011 3019 8 6114 3556 6026 5954 16 11560 7051 12216 12132 32 23181 13079 23449 23561 64 43607 26983 43997 45636 128 80723 45169 81515 81911 256 110925 90185 107332 114046 512 119354 109817 110287 117506 768 112435 105795 106853 111605 1024 107554 105541 105942 109370 2048 88552 79024 80699 90555 4096 61323 54814 58704 61743
clients HEAD | WAL DIO | WAL DIO & WAL BUFFERS READ | WAL BUFFERS READ
1 1404 1070 1424 1375
2 1487 796 1454 1517
4 3064 1743 3011 3019
8 6114 3556 6026 5954
16 11560 7051 12216 12132
32 23181 13079 23449 23561
64 43607 26983 43997 45636
128 80723 45169 81515 81911
256 110925 90185 107332 114046
512 119354 109817 110287 117506
768 112435 105795 106853 111605
1024 107554 105541 105942 109370
2048 88552 79024 80699 90555
4096 61323 54814 58704 61743

--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

On Wed, Sep 24, 2025 at 12:32 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:

On Wed, 2025-09-24 at 07:26 -0700, Bharath Rupireddy wrote:

Right. Reading unflushed WAL buffers for replication was one of the
motivations. But, in general, WALReadFromBuffers has more benefits
since it lets WAL buffers act as a cache for reads, avoiding the need
to re-read WAL from disk for (both physical and logical) replication.
For example, it makes the use of direct I/O for WAL more realistic
and
can provide significant performance benefits [1].

Thanks for looking into this. I did performance analysis with WAL directo I/O to see how reading from WAL buffers affects walsenders: /messages/by-id/CALj2ACV6rS+7iZx5+oAvyXJaN4AG-djAQeM1mrM=YSDkVrUs7g@mail.gmail.com. Following is from that thread. Please let me know if you have any specific cases in mind. I'm happy to run the same test for logical replication.

It helps WAL DIO; since there's no OS
page cache, using WAL buffers as read cache helps a lot. It is clearly
evident from my experiment with WAL DIO patch [1], see the results [2]
and attached graph. As expected, WAL DIO brings down the TPS, whereas
WAL buffers read i.e. this patch brings it up.

[2] Test case is an insert pgbench workload.
clients HEAD | WAL DIO | WAL DIO & WAL BUFFERS READ | WAL BUFFERS READ
1 1404 1070 1424 1375
2 1487 796 1454 1517
4 3064 1743 3011 3019
8 6114 3556 6026 5954
16 11560 7051 12216 12132
32 23181 13079 23449 23561
64 43607 26983 43997 45636
128 80723 45169 81515 81911
256 110925 90185 107332 114046
512 119354 109817 110287 117506
768 112435 105795 106853 111605
1024 107554 105541 105942 109370
2048 88552 79024 80699 90555
4096 61323 54814 58704 61743

Thank you all for reviewing this. Please find the attached rebased
patch for further review.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Hi,

On Sat, Sep 20, 2025 at 1:33 AM Jingtang Zhang <mrdrivingduck@gmail.com> wrote:

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files. The no-locks design ensures it
never gets in the way of backends generating WAL, which is critical
for overall system performance.

Yes, it is actually an interesting thing, beyond current topic. Since
we are using buffered I/O, even though we cannot read from WAL buffer due
to the opportunistic AdvanceXLInsertBuffer by WAL writer, later WALRead may
still find the page inside OS page cache, with high probability, because the
page has just been written out. So WALRead will be fast, too.

When there are logical replication (or logical decoding) clients that
are closely following the WAL insertion point (IOW, the consumers are
all well behaving and catching up with the publisher's WAL generation
rate), such walsenders reading from WAL buffers can still avoid
reading from WAL files (which can either be OS page cache reads or
disk reads) even before the WAL writer's opportunistic WAL buffer
initialization zeros out the WAL data. So it all depends on the
workload and how well the logical replication clients are catching up.

But if we are moving forward to direct I/O some day in the future, the cost
of WALReadFromBuffers and WALRead might be obvious.

Yes, that's correct. I did a quick experiment and will send more details soon.

Maybe the opportunistic
WAL buffer initialization could keep a small ratio of old pages inside WAL
buffer so these pages can still be hit by WALReadFromBuffers.

This is an interesting idea. It could be discussed separately as
slowing down the WAL writer's opportunistic WAL buffer initialization
could have an impact on foreground write workloads.

I rebased and attached the v3 patch.

The v3 patch LGTM.

Thanks for reviewing the patch. I will send the new patch with a fix
that I found in testing soon.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Hi,

On Mon, Sep 22, 2025 at 8:26 AM Jeff Davis <pgsql@j-davis.com> wrote:

On Sat, 2025-09-13 at 22:04 -0700, Bharath Rupireddy wrote:

Thanks for looking at this. Yes, the WAL writers can zero out flushed
buffers before WALReadFromBuffers gets to them. However,
WALReadFromBuffers was intentionally designed as an opportunistic
optimization - it's a "try this first, quickly" approach before
falling back to reading from WAL files.

IIRC, one motivation (perhaps the primary motivation?) was to make it
possible to read buffers before they are flushed. It was always
possible to read already-flushed buffers.

The benefit of reading unflushed buffers is that we can replicate the
WAL sooner (though it can't be replayed until the primary flushes it).
Is that right?

Thanks for taking a look at it. Yes, that's correct.

In general, I would like to reiterate the benefit of reading WAL from
WAL buffers not just for logical walsenders but physical walsenders as
well:

1/ Helps avoid reading from WAL files (which can either be OS page
cache reads or disk reads) whenever possible.

2/ Helps make it more realistic to use direct IO for WAL.
/messages/by-id/20230114203403.4zpg72fw2qb34awf@awork3.anarazel.de

3/ Helps implement the ability to stream out WAL before it has been
locally written out and flushed to improve synchronous replication
performance. I have plans to work on this feature for PG20.
/messages/by-id/20230125211540.zylu74dj2uuh3k7w@awork3.anarazel.de

However, particularly for this patch, I would like to enable the
logical walsenders also to read WAL from WAL buffers when possible.

I did a quick experiment with [1]Test setup: two Amazon EC2 r7i.4xlarge instances (16 vCPU, 128 GB RAM) in the same AZ, one publisher and one subscriber, pg_wal on a dedicated gp3 disk. Publisher runs an insert-only pgbench workload (16 clients) into a two-column table (bigint, text), each insert writing 400 bytes, no indexes. Subscriber tails closely (lag in KB) so the WAL the walsender reads is still in the 2 GB wal_buffers. wal_buffers=2GB, debug_io_direct='wal'. and the following are the results.

With WAL direct IO on, the patch reduces the walsender's WAL reads
from 2.3 GB to 3 MB per run, removing 17 MB/s of physical disk reads
and improving publisher throughput by about 10% (6,701 to 7,361 TPS).
The throughput gain comes from eliminating the WAL read IO on disk, so
WAL writes no longer compete with WAL reads for disk IO. With WAL
direct IO off, the same reads are eliminated at the syscall level with
no throughput change, so it never regresses.

# build WAL direct IO TPS walsender read MB WAL-disk
reads WAL-disk writes replication lag
1 HEAD on 6,701 2,332 17.1 MB/s
16.3 MB/s 23 KB
2 PATCHED on 7,361 3.0 0
17.4 MB/s 14 KB
3 HEAD off 7,672 2,715 0
17.8 MB/s 13 KB
4 PATCHED off 7,694 2.9 0
17.8 MB/s 14 KB

Please find the attached v5 patch. Thanks.

[1]: Test setup: two Amazon EC2 r7i.4xlarge instances (16 vCPU, 128 GB RAM) in the same AZ, one publisher and one subscriber, pg_wal on a dedicated gp3 disk. Publisher runs an insert-only pgbench workload (16 clients) into a two-column table (bigint, text), each insert writing 400 bytes, no indexes. Subscriber tails closely (lag in KB) so the WAL the walsender reads is still in the 2 GB wal_buffers. wal_buffers=2GB, debug_io_direct='wal'.
Test setup: two Amazon EC2 r7i.4xlarge instances (16 vCPU, 128 GB RAM)
in the same AZ, one publisher and one subscriber, pg_wal on a
dedicated gp3 disk. Publisher runs an insert-only pgbench workload (16
clients) into a two-column table (bigint, text), each insert writing
400 bytes, no indexes. Subscriber tails closely (lag in KB) so the WAL
the walsender reads is still in the 2 GB wal_buffers. wal_buffers=2GB,
debug_io_direct='wal'.

Metrics: TPS is publisher pgbench insert throughput. Walsender read MB
is read_bytes from pg_stat_io for the walsender (bytes read through
WALRead()). WAL-disk reads and WAL-disk writes are peak throughput
from iostat on the pg_wal disk. Replication lag is
pg_stat_replication, sampled every 5 seconds, taken as the max over
the run.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: tested, passed
Spec compliant: tested, passed
Documentation: tested, passed

So,the patch applied cleanly to master branch(basically,i cut a branch and applied the patch), builds without issues, and passes the regression tests.

Since,this was my first time reviewing a code in particular, learning from this how buffer-only reads bypass WALRead() and why closing the open segment file descriptor is required to avoid stale descriptors across WAL segment boundaries was a great insight into how WAL reading works.

overall,it looks good

On Mon, Aug 17, 2026 at 04:30:00PM -0700, Bharath Rupireddy wrote:

Test setup: two Amazon EC2 r7i.4xlarge instances (16 vCPU, 128 GB RAM)
in the same AZ, one publisher and one subscriber, pg_wal on a
dedicated gp3 disk. Publisher runs an insert-only pgbench workload (16
clients) into a two-column table (bigint, text), each insert writing
400 bytes, no indexes. Subscriber tails closely (lag in KB) so the WAL
the walsender reads is still in the 2 GB wal_buffers. wal_buffers=2GB,
debug_io_direct='wal'.

Metrics: TPS is publisher pgbench insert throughput. Walsender read MB
is read_bytes from pg_stat_io for the walsender (bytes read through
WALRead()). WAL-disk reads and WAL-disk writes are peak throughput
from iostat on the pg_wal disk. Replication lag is
pg_stat_replication, sampled every 5 seconds, taken as the max over
the run.

Worth noting something in the patch: read_local_xlog_page_guts() is
touched, being called in read_local_xlog_page().
read_local_xlog_page() is used in much more contexts than just the
logical paths and pg_walinspect you are referring to at the top of
this thread. Repack workers, 2PC code, WAL summarizer have also
references to it in their XL_ROUTINE().

This uses pgbench for the WAL inserts. For the logical path, at
least, could a workload based on logical WAL messages generated by
pg_logical_emit_message() be a fancier (aka less noisy) workload to
use to compare the modes of debug_io_direct for the scope of this
patch?

I have to admit that I would be a bit stressed with changing the three
code paths for logical_read_xlog_page(), XLogSendPhysical() and
read_local_xlog_page_guts() all at once.. Checking things separately
seems like a more safer approach, because less risky in terms of
potential reverts of one part of the other if the buildfarm detects
that something is wrong, particularly on slower machines where race
condition patterns show up more easily. My 2c.
--
Michael

Hi,

On Wed, Sep 2, 2026 at 9:08 PM Michael Paquier <michael@paquier.xyz> wrote:

On Mon, Aug 17, 2026 at 04:30:00PM -0700, Bharath Rupireddy wrote:

Test setup: two Amazon EC2 r7i.4xlarge instances (16 vCPU, 128 GB RAM)
in the same AZ, one publisher and one subscriber, pg_wal on a
dedicated gp3 disk. Publisher runs an insert-only pgbench workload (16
clients) into a two-column table (bigint, text), each insert writing
400 bytes, no indexes. Subscriber tails closely (lag in KB) so the WAL
the walsender reads is still in the 2 GB wal_buffers. wal_buffers=2GB,
debug_io_direct='wal'.

Metrics: TPS is publisher pgbench insert throughput. Walsender read MB
is read_bytes from pg_stat_io for the walsender (bytes read through
WALRead()). WAL-disk reads and WAL-disk writes are peak throughput
from iostat on the pg_wal disk. Replication lag is
pg_stat_replication, sampled every 5 seconds, taken as the max over
the run.

Worth noting something in the patch: read_local_xlog_page_guts() is
touched, being called in read_local_xlog_page().
read_local_xlog_page() is used in much more contexts than just the
logical paths and pg_walinspect you are referring to at the top of
this thread. Repack workers, 2PC code, WAL summarizer have also
references to it in their XL_ROUTINE().

Yes, repack and 2PC are covered by the 0002 patch. But I haven't yet
used it for the WAL summarizer. Would it be okay if I do some testing
with the WAL summarizer and propose it as a follow-up patch?

This uses pgbench for the WAL inserts. For the logical path, at
least, could a workload based on logical WAL messages generated by
pg_logical_emit_message() be a fancier (aka less noisy) workload to
use to compare the modes of debug_io_direct for the scope of this
patch?

Thanks for the suggestion. Done. I used pg_logical_emit_message() to
emit the WAL plus pg_recvlogical to let a walsender read it, and here
are the results [1]# GUCs shared_buffers = 8GB max_wal_size = 64GB synchronous_commit = on wal_buffers = 16MB.

With WAL direct IO on, the patch reduces the walsender's WAL reads
from 1.0 GB to 1.6 MB per run, removing 14.5 MB/s of physical disk
reads and improving publisher throughput by about 17% (6,393 to 7,489
TPS). The throughput gain comes from eliminating the WAL read IO on
disk, so WAL writes no longer compete with WAL reads for disk IO. With
WAL direct IO off, the same reads are eliminated at the syscall level
with no throughput change, so it never regresses.

# build WAL direct IO TPS WAL generated MB walsender
read MB WAL-disk reads WAL-disk writes replication lag
1 HEAD on 6,393 511 1,055
14.5 MB/s 9.0 MB/s 0 KB
2 PATCHED on 7,489 598 1.6
0.01 MB/s 10.6 MB/s 0 KB
3 HEAD off 7,347 587 1,229
0 10.4 MB/s 0 KB
4 PATCHED off 7,352 587 1.8
0 10.4 MB/s 0 KB

I have to admit that I would be a bit stressed with changing the three
code paths for logical_read_xlog_page(), XLogSendPhysical() and
read_local_xlog_page_guts() all at once.. Checking things separately
seems like a more safer approach, because less risky in terms of
potential reverts of one part of the other if the buildfarm detects
that something is wrong, particularly on slower machines where race
condition patterns show up more easily. My 2c.

The physical walsender change was supposed to be purely mechanical,
but upon thinking more, I would rather not touch that part. Also, the
way I had it, it had a bug where it ignored the retry part. Sorry
about that.

I split the patches into two. 0001 is for logical walsender, 0002 is
for local WAL reads. 0003 adds a TAP test using an injection point for
the segment boundary issue handled in both patches (I don't intend to
get this committed unless anyone thinks otherwise). A read fully
satisfied from WAL buffers can leave the segment file open on the
wrong segment. This is handled by closing the open segment after such
a read so the next file read reopens the correct one.

PS: There is an opportunity to deduplicate with a wrapper function on
the code that 0001 and 0002 add for WALReadFromBuffers()+WALRead(). I
chose not to add that wrapper, just because we cannot reuse it in the
physical walsenders, defeating the purpose of deduplication. I am open
to thoughts on this.

Please find the attached v6 patches.

[1]: # GUCs shared_buffers = 8GB max_wal_size = 64GB synchronous_commit = on wal_buffers = 16MB
# GUCs
shared_buffers = 8GB
max_wal_size = 64GB
synchronous_commit = on
wal_buffers = 16MB

# Session 1
psql -c "SELECT pg_create_logical_replication_slot('bench','test_decoding');"
printf "SELECT pg_logical_emit_message(true, 'bench', repeat('x',
350));\n" > emit.sql
pgbench -n -f emit.sql -c 16 -j 16 -T 180 postgres

pg_recvlogical -p 5432 -d postgres --slot bench --start -f /dev/null &

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com