enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

Started by Jakub Wartak2 months ago11 messageshackers

jakub.wartak@enterprisedb.com

2 months ago

Hi,

While investigating pg_basebackup performance, I found that a significant
amount of CPU is being wasted on both the server and client sides, and that
there are substantial opportunities to improve throughput without
fundamentally changing the design (so without abandoning the simple
single-threaded, single-connection design). Keeping that design intact also
lets us answer some of the questions raised in the old parallel-backup thread
[1]: /messages/by-id/20200420201922.55ab7ovg6535suyz@alap3.anarazel.de
easier (0001-0003), some small optimizations that make a significant
difference in my production-like testing (0004-0008), and then implement
io_uring (Direct I/O + Async I/O) support for pg_basebackup (0009-0010).

First, a quick performance demonstration, from the AWS benchmarks described in
full later (2x c6in.8xlarge, specs in [4]HW: 2x c6in.8xlarge EC2 (each with 32 vCPUs, 64 GB RAM), scale 4000 / 50 GB cluster size, 3x NVMe of 50 GB / 8k IOPS each in LVM/RAID0 (lvcreate -i 3), ext4, tuned TCP stack (BBR), ENA Express enabled. In theory and sometimes in practice this gives a real ~23 Gbps, sometimes more like 15 Gbps -- highly nondeterministic and dependent on zone, region, and time of day. Aggregate ~50 Gbps bandwidth was always available without problems; only the single-flow TCP performance fluctuated, typically between 15 and 19 Gbps during these measurements, and I rather lost hope of it being deterministic. It seems to be a function of the env rather than the configuration, which was identical throughout the excercies. I suspect that on raw hardware with >= 40 GbE interfaces higher speeds are possible, but the intent here was to simulate what real customers actually see.), when taking a backup without
checksum verification/SSL, while still using single socket/fd:

master + 0001-0003, writing to disk: ~1610MB/s.
master + all patches, with MPTCP: 2750-2975MB/s.

That's like ~1.85x speedup on real cloud hardware. What I found is that the
gains come from several independent sources: reducing syscall numbers,
avoiding redundant memory copies in libpq, and using Direct I/O with
asynchronous submission on the client side. Above such rates, I believe we
genuinely need independent connections and a parallel-backup design, but
everything up to that point is achievable with relatively low-invasive
changes.

OK, now let's go through the patches:

0001 and 0002 extend the existing server-side --target blackhole concept into
separate server and client (pg_basebackup) blackholes, so that each stage of
the data transfer pipeline can be measured reliably. Together with 0003, this
gives DBAs a simple way to locate a bottleneck using nothing but the tool
itself:

-t server-blackhole measures the server's raw read speed
-t client-blackhole additionally sends the data over the network and but
discard writes; and a plain(classic) -Fp -D /path run exposes all three
potential bottlenecks at once.

0003 adds a simple timing message with the average transfer rate. The idea is
that if we're going to discuss numbers on this thread, it's better if the tool
measures them the same way for everyone, rather than each of us relying on
atop or other tooling.

0004 increases SINK_BUFFER_LENGTH from its current 32 kB. Profiling the
server side shows a huge number of small pread() calls coming from
basebackup_read_file(), which is surprising given that elsewhere in the tree
(backend/storage/buffer/README) we already mention that a 256 kB ring for seq.
scans scans is used because it fits comfortably in L2 cache. On my laptop,
with no SSL, no checksum generation or verification, a hot filesystem cache,
and a client blackhole to eliminate client I/O, a 10 GB backup over loopback
runs at:
3.1 GB/s with the 32 kB buffer,
5.1 GB/s at 128 kB,
5.5 GB/s at 256 kB,
6.0 GB/s at 1 MB
(those are average of five runs). The win comes from letting pread()
swallow much larger chunks of each segment in one go: per-core L2 caches are
1-2 MB even on laptops these days, syscalls have become more expensive since
the Spectre/Meltdown mitigations, and the kernel's default readahead is
already in the 128-512 kB range, so there's little reason to trickle the data
through 32 kB at a time. I also experimented with raising PQ_SEND_BUFFER_SIZE
from 8 kB to 128 kB, but saw no improvement and sometimes a regression --
apparently it defeats libpq's direct-send optimization for large messages.

0005 issues posix_fadvise(POSIX_FADV_SEQUENTIAL) on the assumption that
segments are usually cold at backup time. Even with 0004 applied, cold-cache
throughput on my laptop's NVMe goes from 1.5 GB/s to 2 GB/s with this patch,
because the synchronous pread() calls see lower latencies (visible with eBPF
funclatency). One question I tried to answer was whether it makes sense to
issue one fadvise call per file or many smaller ones (in the spirit of
maintenance_io_concurrency), but on Linux FADV_SEQUENTIAL does exactly one
thing -- it just widens the maximum readahead horizon -- so repeated calls buy
nothing; it's simply handled with read-as-you-go by the kernel's own readahead
heuristics.

0006 avoids buffering client-side writes with glibc stdio (FILE *). I
initially thought this is going to be as easy as a setvbuf(3) call, since the
extractor path was issuing a mismatched 4 kB + 258 kB write pair per 256 kB
received. A 1MB setvbuf buffer produced the tidy syscall pattern I wanted, but
it actually got slower when measured carefully on tmpfs: glibc's fwrite()
internally stats the file for its block size and introduces an extra memcpy()
for large writes. Direct I/O would bypass this entirely, but that felt
premature at this stage, so the patch takes a more conservative approach.
Throughput is roughly unchanged (3.1 vs 3.2 GB/s to a ramdisk), but the
syscall pattern becomes a clean one-to-one recvfrom()/write() pairing, which
matters for the later patches.

0007 preallocates output files via posix_fallocate(). This gave about 7%
essentially for free, though you need to already be writing fast (probably
more than of 1 GB/s) before it becomes visible.

0008 eliminates a redundant memory copy in the receive path. After the fixes
above, perf showed PQgetCopyData()'s internal memcpy() consuming 60-70% of
pg_basebackup's single CPU: libpq allocates a buffer and copies the incoming
data into it, only for the callback (ReceiveArchiveStreamChunk and friends) to
immediately consume that same buffer at an offset of one byte, skipping the
protocol message byte. Since pg_basebackup is the only consumer here, the copy
can simply be avoided. On the receive path the kernel already copies from the
NIC via DMA to a kernel buffer and again into userspace, so above roughly 10
Gbps this extra PostgreSQL-side copy becomes very visible. With the copy
removed, a CPU-saturated pg_basebackup over localhost goes from 2.4 GB/s to
~5.5 GB/s when not writing to disk; when writing even to a ramdisk, the
bottleneck shifts to fundamental kernel-side vfs_write() costs such as cgroups
v2 memory accounting.

0009: Even with all of the above, there was still idle sequential write
bandwidth left on the table, and buffered writes couldn't reach it (without
some parallelism). Direct I/O alone (O_DIRECT) carried an unavoidable latency
hit and wouldn't be usable here alone either, so it had be combined with
asynchronous submission, which means liburing (client-side only). So this
patch adds io_uring (Direct I/O) for larger files on the client side. The
patch adds PG_BASEBACKUP_NODIO too for experimenting with this (set/unset it
to see the difference).

0010 fixes a serious regression that 0009 introduced for -Ft (tar) output.
Because the final archive length isn't known in advance -- unlike -Fp (plain),
where each file's target size is known -- every Direct I/O write via io_uring
also extended the file, triggering synchronous space allocation and
serializing on ext4's per-inode i_rwsem, which defeated the entire point of a
deep async queue. The fix is proper posix_fallocate() for the tar path, with a
fallback to buffered writes when preallocation isn't possible. It's a separate
commit because it needed its own explanation: without it, tar output ran at
410MB/s; with it, ~2400MB/s.

Now, the real AWS benchmarks. These are averages of 3-5 runs with the
lowest outliers discarded, on 2x c6in.8xlarge with ENA Express (full specs in
[4]: HW: 2x c6in.8xlarge EC2 (each with 32 vCPUs, 64 GB RAM), scale 4000 / 50 GB cluster size, 3x NVMe of 50 GB / 8k IOPS each in LVM/RAID0 (lvcreate -i 3), ext4, tuned TCP stack (BBR), ENA Express enabled. In theory and sometimes in practice this gives a real ~23 Gbps, sometimes more like 15 Gbps -- highly nondeterministic and dependent on zone, region, and time of day. Aggregate ~50 Gbps bandwidth was always available without problems; only the single-flow TCP performance fluctuated, typically between 15 and 19 Gbps during these measurements, and I rather lost hope of it being deterministic. It seems to be a function of the env rather than the configuration, which was identical throughout the excercies. I suspect that on raw hardware with >= 40 GbE interfaces higher speeds are possible, but the intent here was to simulate what real customers actually see.
unless stated otherwise:

stageI: hw baseline
===================
a1. master + 000[123] -t server-blackhole # 11573MB/s (disk I/O possible
on serverside from hot pagecache)
a2. master + 000[123] -t client-blackhole # 1820MB/s (single-thread TCP limit,
no writing client side)
a3. master + 000[123] -D /db/backup # ~1610MB/s (when writing, proper backup)

stageII: basic optimizations
============================

+0004 SINK_BUFFER_LENGTH increase:
b. master + 000[1234] -t client-blackhole # 2158MB/s

cold-cache scenarios (how efficient we are when data is not in the VFS cache
server-side; echo 3 > drop_caches):
c1. master + 000[1234] -t client-blackhole # 1476MB/s
+ 0005 posix_fadvise whole file:
c2. master + 000[12345] -t client-blackhole # 1610MB/s

+0006, hot-cache scenarios again, real I/O by pg_basebackup, 100% CPU
throughout for the tests below:
d1. master + 000[12345] -D /db/backup # ~1639MB/s
d2. master + 000[123456] -D /db/backup # around the same, but we now issue
matched recvfrom()+write() pairs

+0007 preallocate file
e. master + 000[1234567] -D /db/backup # ~1687MB/s

+0008 avoid the second memcpy in libpq
f. master + 000[12345678] -D /db/backup # 1818MB/s (we cannot get more;
this saturates the AWS's ENA Express link on a single connection/
technically maxes out AWS network fabric/SDN)

crosscheck against baseline, just in case:
g. master + 000[123] -D /db/backup # 1613MB/s

stageIII: MPTCP
===============
I had to fall back to MPTCP [2]/messages/by-id/CAKZiRmy6j9PBzDHZwdgwHavwKDzv5GWtRSWOTj6-jv6SCOZ=YA@mail.gmail.com because the maximum single-stream TCP
bandwidth even with ENA Express fluctuated far too much to isolate. That patch
is going to posted independently in its own thread [1]/messages/by-id/20200420201922.55ab7ovg6535suyz@alap3.anarazel.de, but attached here too.

h0. iperf3 max with MPTCP, single connection: 43Gbit/s (but fluctuating down
to just 27Gbit/s sometimes; just 2-3 subflows)
h1. master + 000[123] + PGMPTCP=0 -t client-blackhole # 2079MB/s
h2. master + 000[123] + PGMPTCP=1 -t client-blackhole # 2770MB/s
h3. master + 000[123] + PGMPTCP=0 -D /db/backup # 1870MB/s, occasionally
slightly lower (ENA Express)
h4. master + 000[12345678] + PGMPTCP=1 -t client-blackhole # 3840MB/s, a
real indication that the earlier ceiling was just AWS network --
though note we are not actually writing here

stageIV: DIO+AIO
================
Because pg_basebackup was pinned at 100% CPU while the storage could still
take more, we need io_uring with DIO+AIO (+0009):

i1. master + 000[123456789] + PGMPTCP=1 -D /db/backup # 2750-2975MB/s; we
max out MPTCP bandwidth with the core saturated by writing
i2. master + 000[123456789] + PGMPTCP=0 -D /db/backup # 1695MB/s, a sample
of the fluctuations (down from 1870MB/s)
i3. master + 000[123456789] + PGMPTCP=1 without --no-verify-checksums
# still 2772MB/s (server-side checksum calculation is fast enough!)
i4. master + 000[123456789] + PGMPTCP=1 without --no-verify-checksums and
with default CRC32C for manifest checksumming # still 2673MB/s

So "i4." divided by "a3." gives 2975MB/ 1610 = ~1.85x , with some of the 0009
benefits greatly take from ideas in 0007 and 0008.

There were several surprises in the above, at least to me -- I hadn't
anticipated hardware-assisted checksums being that fast.

Tar (-Ft) output initially suffered badly from the synchronous space
allocation (because that was not done), as per details described under 0010.
With proper posix_fallocate() it recovers performance. Without checksums and
without SSL:
j1. master + 000[123456789] + PGMPTCP=1 -D /db/backup.tar -Ft # 410MB/s,
regression due to the lack of effective posix_fallocate()
j2. master + 000[123456789]+0010 + PGMPTCP=1 -D /db/backup.tar -Ft
# ~2400MB/s

TLS
===
Some measurements of TLS itself in this setup (TLSv1.3, cipher
TLS_AES_256_GCM_SHA384):
k1. ssl=on + master + 000[123] + PGMPTCP=1 -D /db/backup
#baseline of ~1247MB/s
k2. ssl=on + master + 000[123456789] + PGMPTCP=1 -D /db/backup
#1668-1791MB/s

On SSL specifically: the cipher alone can push close to 8-10 GB/s (64-80
Gbps) on a single CPU with openssl -bench, and reducing the number of
encryption rounds doesn't appear to yield any real benefit. But since SSL/TLS
is additional work performed on the __same__ thread, it imposes a clear
CPU overhead here: 1791 / 2975 = 0.60x.

A few conclusions:

- pg_basebackup today tops out at roughly 10-15 Gbps, because in the end we
are constrained by what a single core can do.

- pg_basebackup is very vulnerable to single-TCP-flow performance on real
networks. These tests were intentionally run on real hardware and networks
within a single zone -- the kind people actually use -- rather than on an
isolated network. See table [3]a small audit of the single-TCP-connection speeds one can expect depending on where one is running: for apparent real-world limitations;
single-flow TCP is IMHO the top constraint today, often well below 10-25
Gbps, and without MPTCP there is no simple way to get past it...

- Some tuning is possible on our side without invasive protocol changes.

- MPTCP is easy to set up and enabled almost everywhere today, but it has
its own ceiling: apparently ~40 Gbps in iperf3 and realistically ~25 Gbps
when we are also writing single-threaded. Potentially in far future we
could put recvfrom() (where MPTCP reassembly happens) on one thread and
issue SQs/io_uring_submit() from a second thread, but that would require
Thomas's pg_thr_*() APIs to be in place.

- I researched SSL/kTLS/sendfile/zcrx a bit as well, but this email is
already too long, so that's not covered here.

Based on the measurements and conclusions above, I think we can improve
pg_basebackup's single-threaded performance for now with these
relatively low-invasive patches.

Thanks for reading that far.

-J.

[1]: /messages/by-id/20200420201922.55ab7ovg6535suyz@alap3.anarazel.de
[2]: /messages/by-id/CAKZiRmy6j9PBzDHZwdgwHavwKDzv5GWtRSWOTj6-jv6SCOZ=YA@mail.gmail.com
[3]: a small audit of the single-TCP-connection speeds one can expect depending on where one is running:
depending on where one is running:

[4]: HW: 2x c6in.8xlarge EC2 (each with 32 vCPUs, 64 GB RAM), scale 4000 / 50 GB cluster size, 3x NVMe of 50 GB / 8k IOPS each in LVM/RAID0 (lvcreate -i 3), ext4, tuned TCP stack (BBR), ENA Express enabled. In theory and sometimes in practice this gives a real ~23 Gbps, sometimes more like 15 Gbps -- highly nondeterministic and dependent on zone, region, and time of day. Aggregate ~50 Gbps bandwidth was always available without problems; only the single-flow TCP performance fluctuated, typically between 15 and 19 Gbps during these measurements, and I rather lost hope of it being deterministic. It seems to be a function of the env rather than the configuration, which was identical throughout the excercies. I suspect that on raw hardware with >= 40 GbE interfaces higher speeds are possible, but the intent here was to simulate what real customers actually see.
4000 / 50 GB cluster size, 3x NVMe of 50 GB / 8k IOPS each in LVM/RAID0
(lvcreate -i 3), ext4, tuned TCP stack (BBR), ENA Express enabled. In theory
and sometimes in practice this gives a real ~23 Gbps, sometimes more like
15 Gbps -- highly nondeterministic and dependent on zone, region, and time of
day. Aggregate ~50 Gbps bandwidth was always available without
problems; only the single-flow TCP performance fluctuated, typically between
15 and 19 Gbps during these measurements, and I rather lost hope of it being
deterministic. It seems to be a function of the env rather than the
configuration, which was identical throughout the excercies. I suspect that on
raw hardware with >= 40 GbE interfaces higher speeds are possible, but the
intent here was to simulate what real customers actually see.

sysctls:
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 2097152
net.core.wmem_default = 2097152
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
mptcp enabled by default so no change

Gustavo William

gustavowilliam0805@gmail.com

about 2 months ago

In reply to: Jakub Wartak (#1)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

Hi,

I'm implementing an improvement for incremental backups (.txt patch
attached) which I think is relevant to this thread.

Context: I noticed pg_basebackup incremental backups can sometimes be very
slow, sometimes even slower than a full backup. The more scattered the
changed blocks, the worse it gets. Looking at disk metrics and the code I
found one of the reasons is that delta blocks are requested serially, each
changed block read strictly one at a time in a loop.

A simple fix, while still keeping the single-threaded design, was to issue
posix_fadvise() calls for upcoming blocks before the code gets to them, in
the hope that the kernel brings them into cache before they're actually
read.

A few results so far, on a ~51GB cluster:
~10 min of pgbench running, then taking an incremental backup of 28GB took
2min 51s (~167.7 MiB/s). With the patched code: 2min 18s (~207.8 MiB/s),
~24% higher.
With more scattered changes (via TABLESAMPLE SYSTEM(20), touching ~20% of a
table uniformly at random): 31GB incremental in 4min 22s (~121.2 MiB/s).
With the patched code: 2min 32s (~208.8 MiB/s), ~72% higher.

Short overview of the patch:
1. Before iterating over the delta blocks, fadvise the first N blocks.
2. After each iteration (each block read), fadvise the block at
positioncurrent_block + N.
3. Repeat this every iteration until the end.

This way, we're always at least N blocks ahead of the current one. For now,
N = maintenance_io_concurrency.

The logic is similar to patch 0005 from Jakub, but for the random read
pattern instead.

I intend to keep working on this in a couple of days (add tests, collect
metrics), but wanted to share where it stands now in case anyone has
thoughts in the meantime.

Thanks,
Gustavo Oliveira

Em sex., 28 de ago. de 2026 às 17:16, Jakub Wartak <
jakub.wartak@enterprisedb.com> escreveu:

Show quoted text

Hi,

While investigating pg_basebackup performance, I found that a significant
amount of CPU is being wasted on both the server and client sides, and that
there are substantial opportunities to improve throughput without
fundamentally changing the design (so without abandoning the simple
single-threaded, single-connection design). Keeping that design intact also
lets us answer some of the questions raised in the old parallel-backup
thread
[1]. The attached patches implement a couple of changes to make
benchmarking
easier (0001-0003), some small optimizations that make a significant
difference in my production-like testing (0004-0008), and then implement
io_uring (Direct I/O + Async I/O) support for pg_basebackup (0009-0010).

First, a quick performance demonstration, from the AWS benchmarks
described in
full later (2x c6in.8xlarge, specs in [4]), when taking a backup without
checksum verification/SSL, while still using single socket/fd:

master + 0001-0003, writing to disk: ~1610MB/s.
master + all patches, with MPTCP: 2750-2975MB/s.

That's like ~1.85x speedup on real cloud hardware. What I found is that the
gains come from several independent sources: reducing syscall numbers,
avoiding redundant memory copies in libpq, and using Direct I/O with
asynchronous submission on the client side. Above such rates, I believe we
genuinely need independent connections and a parallel-backup design, but
everything up to that point is achievable with relatively low-invasive
changes.

OK, now let's go through the patches:

0001 and 0002 extend the existing server-side --target blackhole concept
into
separate server and client (pg_basebackup) blackholes, so that each stage
of
the data transfer pipeline can be measured reliably. Together with 0003,
this
gives DBAs a simple way to locate a bottleneck using nothing but the tool
itself:

-t server-blackhole measures the server's raw read speed
-t client-blackhole additionally sends the data over the network and but
discard writes; and a plain(classic) -Fp -D /path run exposes all three
potential bottlenecks at once.

0003 adds a simple timing message with the average transfer rate. The idea
is
that if we're going to discuss numbers on this thread, it's better if the
tool
measures them the same way for everyone, rather than each of us relying on
atop or other tooling.

0004 increases SINK_BUFFER_LENGTH from its current 32 kB. Profiling the
server side shows a huge number of small pread() calls coming from
basebackup_read_file(), which is surprising given that elsewhere in the
tree
(backend/storage/buffer/README) we already mention that a 256 kB ring for
seq.
scans scans is used because it fits comfortably in L2 cache. On my laptop,
with no SSL, no checksum generation or verification, a hot filesystem
cache,
and a client blackhole to eliminate client I/O, a 10 GB backup over
loopback
runs at:
3.1 GB/s with the 32 kB buffer,
5.1 GB/s at 128 kB,
5.5 GB/s at 256 kB,
6.0 GB/s at 1 MB
(those are average of five runs). The win comes from letting pread()
swallow much larger chunks of each segment in one go: per-core L2 caches
are
1-2 MB even on laptops these days, syscalls have become more expensive
since
the Spectre/Meltdown mitigations, and the kernel's default readahead is
already in the 128-512 kB range, so there's little reason to trickle the
data
through 32 kB at a time. I also experimented with raising
PQ_SEND_BUFFER_SIZE
from 8 kB to 128 kB, but saw no improvement and sometimes a regression --
apparently it defeats libpq's direct-send optimization for large messages.

0005 issues posix_fadvise(POSIX_FADV_SEQUENTIAL) on the assumption that
segments are usually cold at backup time. Even with 0004 applied,
cold-cache
throughput on my laptop's NVMe goes from 1.5 GB/s to 2 GB/s with this
patch,
because the synchronous pread() calls see lower latencies (visible with
eBPF
funclatency). One question I tried to answer was whether it makes sense to
issue one fadvise call per file or many smaller ones (in the spirit of
maintenance_io_concurrency), but on Linux FADV_SEQUENTIAL does exactly one
thing -- it just widens the maximum readahead horizon -- so repeated calls
buy
nothing; it's simply handled with read-as-you-go by the kernel's own
readahead
heuristics.

0006 avoids buffering client-side writes with glibc stdio (FILE *). I
initially thought this is going to be as easy as a setvbuf(3) call, since
the
extractor path was issuing a mismatched 4 kB + 258 kB write pair per 256 kB
received. A 1MB setvbuf buffer produced the tidy syscall pattern I wanted,
but
it actually got slower when measured carefully on tmpfs: glibc's fwrite()
internally stats the file for its block size and introduces an extra
memcpy()
for large writes. Direct I/O would bypass this entirely, but that felt
premature at this stage, so the patch takes a more conservative approach.
Throughput is roughly unchanged (3.1 vs 3.2 GB/s to a ramdisk), but the
syscall pattern becomes a clean one-to-one recvfrom()/write() pairing,
which
matters for the later patches.

0007 preallocates output files via posix_fallocate(). This gave about 7%
essentially for free, though you need to already be writing fast (probably
more than of 1 GB/s) before it becomes visible.

0008 eliminates a redundant memory copy in the receive path. After the
fixes
above, perf showed PQgetCopyData()'s internal memcpy() consuming 60-70% of
pg_basebackup's single CPU: libpq allocates a buffer and copies the
incoming
data into it, only for the callback (ReceiveArchiveStreamChunk and
friends) to
immediately consume that same buffer at an offset of one byte, skipping the
protocol message byte. Since pg_basebackup is the only consumer here, the
copy
can simply be avoided. On the receive path the kernel already copies from
the
NIC via DMA to a kernel buffer and again into userspace, so above roughly
10
Gbps this extra PostgreSQL-side copy becomes very visible. With the copy
removed, a CPU-saturated pg_basebackup over localhost goes from 2.4 GB/s to
~5.5 GB/s when not writing to disk; when writing even to a ramdisk, the
bottleneck shifts to fundamental kernel-side vfs_write() costs such as
cgroups
v2 memory accounting.

0009: Even with all of the above, there was still idle sequential write
bandwidth left on the table, and buffered writes couldn't reach it (without
some parallelism). Direct I/O alone (O_DIRECT) carried an unavoidable
latency
hit and wouldn't be usable here alone either, so it had be combined with
asynchronous submission, which means liburing (client-side only). So this
patch adds io_uring (Direct I/O) for larger files on the client side. The
patch adds PG_BASEBACKUP_NODIO too for experimenting with this (set/unset
it
to see the difference).

0010 fixes a serious regression that 0009 introduced for -Ft (tar) output.
Because the final archive length isn't known in advance -- unlike -Fp
(plain),
where each file's target size is known -- every Direct I/O write via
io_uring
also extended the file, triggering synchronous space allocation and
serializing on ext4's per-inode i_rwsem, which defeated the entire point
of a
deep async queue. The fix is proper posix_fallocate() for the tar path,
with a
fallback to buffered writes when preallocation isn't possible. It's a
separate
commit because it needed its own explanation: without it, tar output ran at
410MB/s; with it, ~2400MB/s.

Now, the real AWS benchmarks. These are averages of 3-5 runs with the
lowest outliers discarded, on 2x c6in.8xlarge with ENA Express (full specs
in
[4]), using --no-verify-checksums and --manifest-checksums=NONE, and no SSL
unless stated otherwise:

stageI: hw baseline
===================
a1. master + 000[123] -t server-blackhole # 11573MB/s (disk I/O possible
on serverside from hot pagecache)
a2. master + 000[123] -t client-blackhole # 1820MB/s (single-thread TCP
limit,
no writing client side)
a3. master + 000[123] -D /db/backup # ~1610MB/s (when writing, proper
backup)

stageII: basic optimizations
============================

+0004 SINK_BUFFER_LENGTH increase:
b. master + 000[1234] -t client-blackhole # 2158MB/s

cold-cache scenarios (how efficient we are when data is not in the VFS
cache
server-side; echo 3 > drop_caches):
c1. master + 000[1234] -t client-blackhole # 1476MB/s
+ 0005 posix_fadvise whole file:
c2. master + 000[12345] -t client-blackhole # 1610MB/s

+0006, hot-cache scenarios again, real I/O by pg_basebackup, 100% CPU
throughout for the tests below:
d1. master + 000[12345] -D /db/backup # ~1639MB/s
d2. master + 000[123456] -D /db/backup # around the same, but we now issue
matched recvfrom()+write() pairs

+0007 preallocate file
e. master + 000[1234567] -D /db/backup # ~1687MB/s

+0008 avoid the second memcpy in libpq
f. master + 000[12345678] -D /db/backup # 1818MB/s (we cannot get more;
this saturates the AWS's ENA Express link on a single connection/
technically maxes out AWS network fabric/SDN)

crosscheck against baseline, just in case:
g. master + 000[123] -D /db/backup # 1613MB/s

stageIII: MPTCP
===============
I had to fall back to MPTCP [2] because the maximum single-stream TCP
bandwidth even with ENA Express fluctuated far too much to isolate. That
patch
is going to posted independently in its own thread [1], but attached here
too.

h0. iperf3 max with MPTCP, single connection: 43Gbit/s (but fluctuating
down
to just 27Gbit/s sometimes; just 2-3 subflows)
h1. master + 000[123] + PGMPTCP=0 -t client-blackhole # 2079MB/s
h2. master + 000[123] + PGMPTCP=1 -t client-blackhole # 2770MB/s
h3. master + 000[123] + PGMPTCP=0 -D /db/backup # 1870MB/s, occasionally
slightly lower (ENA Express)
h4. master + 000[12345678] + PGMPTCP=1 -t client-blackhole # 3840MB/s, a
real indication that the earlier ceiling was just AWS network --
though note we are not actually writing here

stageIV: DIO+AIO
================
Because pg_basebackup was pinned at 100% CPU while the storage could still
take more, we need io_uring with DIO+AIO (+0009):

i1. master + 000[123456789] + PGMPTCP=1 -D /db/backup # 2750-2975MB/s; we
max out MPTCP bandwidth with the core saturated by writing
i2. master + 000[123456789] + PGMPTCP=0 -D /db/backup # 1695MB/s, a sample
of the fluctuations (down from 1870MB/s)
i3. master + 000[123456789] + PGMPTCP=1 without --no-verify-checksums
# still 2772MB/s (server-side checksum calculation is fast enough!)
i4. master + 000[123456789] + PGMPTCP=1 without --no-verify-checksums and
with default CRC32C for manifest checksumming # still 2673MB/s

So "i4." divided by "a3." gives 2975MB/ 1610 = ~1.85x , with some of the
0009
benefits greatly take from ideas in 0007 and 0008.

There were several surprises in the above, at least to me -- I hadn't
anticipated hardware-assisted checksums being that fast.

Tar (-Ft) output initially suffered badly from the synchronous space
allocation (because that was not done), as per details described under
0010.
With proper posix_fallocate() it recovers performance. Without checksums
and
without SSL:
j1. master + 000[123456789] + PGMPTCP=1 -D /db/backup.tar -Ft # 410MB/s,
regression due to the lack of effective posix_fallocate()
j2. master + 000[123456789]+0010 + PGMPTCP=1 -D /db/backup.tar -Ft
# ~2400MB/s

TLS
===
Some measurements of TLS itself in this setup (TLSv1.3, cipher
TLS_AES_256_GCM_SHA384):
k1. ssl=on + master + 000[123] + PGMPTCP=1 -D /db/backup
#baseline of ~1247MB/s
k2. ssl=on + master + 000[123456789] + PGMPTCP=1 -D /db/backup
#1668-1791MB/s

On SSL specifically: the cipher alone can push close to 8-10 GB/s (64-80
Gbps) on a single CPU with openssl -bench, and reducing the number of
encryption rounds doesn't appear to yield any real benefit. But since
SSL/TLS
is additional work performed on the __same__ thread, it imposes a clear
CPU overhead here: 1791 / 2975 = 0.60x.

A few conclusions:

- pg_basebackup today tops out at roughly 10-15 Gbps, because in the end we
are constrained by what a single core can do.

- pg_basebackup is very vulnerable to single-TCP-flow performance on real
networks. These tests were intentionally run on real hardware and
networks
within a single zone -- the kind people actually use -- rather than on an
isolated network. See table [3] for apparent real-world limitations;
single-flow TCP is IMHO the top constraint today, often well below 10-25
Gbps, and without MPTCP there is no simple way to get past it...

- Some tuning is possible on our side without invasive protocol changes.

- MPTCP is easy to set up and enabled almost everywhere today, but it has
its own ceiling: apparently ~40 Gbps in iperf3 and realistically ~25 Gbps
when we are also writing single-threaded. Potentially in far future we
could put recvfrom() (where MPTCP reassembly happens) on one thread and
issue SQs/io_uring_submit() from a second thread, but that would require
Thomas's pg_thr_*() APIs to be in place.

- I researched SSL/kTLS/sendfile/zcrx a bit as well, but this email is
already too long, so that's not covered here.

Based on the measurements and conclusions above, I think we can improve
pg_basebackup's single-threaded performance for now with these
relatively low-invasive patches.

Thanks for reading that far.

-J.

[1] -
/messages/by-id/20200420201922.55ab7ovg6535suyz@alap3.anarazel.de
[2] -
/messages/by-id/CAKZiRmy6j9PBzDHZwdgwHavwKDzv5GWtRSWOTj6-jv6SCOZ=YA@mail.gmail.com
[3] - a small audit of the single-TCP-connection speeds one can expect
depending on where one is running:

Hardware type | Max aggr bw | 1x TCP LAN | 1x TCP MAN| 1x TCP
WAN
Physical server | <= 200 GbE | 1 CPU | 1-10Gbps |
1-10Gbps?
VM (Xen, VMware, etc) | <= 100 GbE? | 1 CPU | 1-10Gbps |
1-10Gbps?
lowend AWS VM | 5-40 Gbps | 5 Gbps | 5 Gbps | <= 5
Gbps
lowend AWS VM Cluster Pl. | <= 200 Gbps | 10 Gbps | 5 Gbps | <= 5
Gbps
highend AWS VM | <= 200 Gbps | 5 Gbps | 5 Gbps | <= 5
Gbps
highend AWS VM+ENA Express| <= 200 Gbps | 25 Gbps | 25 Gbps | <= 5
Gbps
lowend Azure VM | 3(!)-40 Gbps| 3(!) Gbps | 1.5-3 Gbps| 1.5-3
Gbps
highend Azure VM+acc. net.| <= 200 Gbps | 10-12.5Gbps| 10 Gbps | 1.5-3
Gbps

[4] - HW: 2x c6in.8xlarge EC2 (each with 32 vCPUs, 64 GB RAM), scale
4000 / 50 GB cluster size, 3x NVMe of 50 GB / 8k IOPS each in LVM/RAID0
(lvcreate -i 3), ext4, tuned TCP stack (BBR), ENA Express enabled. In
theory
and sometimes in practice this gives a real ~23 Gbps, sometimes more like
15 Gbps -- highly nondeterministic and dependent on zone, region, and time
of
day. Aggregate ~50 Gbps bandwidth was always available without
problems; only the single-flow TCP performance fluctuated, typically
between
15 and 19 Gbps during these measurements, and I rather lost hope of it
being
deterministic. It seems to be a function of the env rather than the
configuration, which was identical throughout the excercies. I suspect
that on
raw hardware with >= 40 GbE interfaces higher speeds are possible, but the
intent here was to simulate what real customers actually see.

sysctls:
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 2097152
net.core.wmem_default = 2097152
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
mptcp enabled by default so no change

Jakub Wartak

jakub.wartak@enterprisedb.com

about 2 months ago

In reply to: Gustavo William (#2)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On Fri, Aug 28, 2026 at 10:44 PM Gustavo William
<gustavowilliam0805@gmail.com> wrote:

Hi,

I'm implementing an improvement for incremental backups (.txt patch attached) which I think is relevant to this thread.

Context: I noticed pg_basebackup incremental backups can sometimes be very slow, sometimes even slower than a full backup. The more scattered the changed blocks, the worse it gets. Looking at disk metrics and the code I found one of the reasons is that delta blocks are requested serially, each changed block read strictly one at a time in a loop.

A simple fix, while still keeping the single-threaded design, was to issue posix_fadvise() calls for upcoming blocks before the code gets to them, in the hope that the kernel brings them into cache before they're actually read.

A few results so far, on a ~51GB cluster:
~10 min of pgbench running, then taking an incremental backup of 28GB took 2min 51s (~167.7 MiB/s). With the patched code: 2min 18s (~207.8 MiB/s), ~24% higher.
With more scattered changes (via TABLESAMPLE SYSTEM(20), touching ~20% of a table uniformly at random): 31GB incremental in 4min 22s (~121.2 MiB/s). With the patched code: 2min 32s (~208.8 MiB/s), ~72% higher.

Short overview of the patch:
1. Before iterating over the delta blocks, fadvise the first N blocks.
2. After each iteration (each block read), fadvise the block at positioncurrent_block + N.
3. Repeat this every iteration until the end.

This way, we're always at least N blocks ahead of the current one. For now, N = maintenance_io_concurrency.

The logic is similar to patch 0005 from Jakub, but for the random read pattern instead.

I intend to keep working on this in a couple of days (add tests, collect metrics), but wanted to share where it stands now in case anyone has thoughts in the meantime.

Hi Gustavo,

I would suggest we merge Your's and mine into one bigger combined 0005 later
unless someone opposes (we would be both coauthors). The reason is that
aparently I've forgotten to support incremental mode (which means that I ended
up issuing posix_fadvise SEQUENTIAL even in incremental mode, which is not
necessary) and Your's introduces concept of advising for only needed blocks
in the incremental path, but first, my review of Your's one:

1. there's bug that when m_io_c = 0 you still issue posix_fadvise()s from 2nd
callsite (no way to turn it off) and also with e.g. m_io_c = 1 we seem to
still issue an additional one.

2. I'm not sure f that merging of adjacent blocks shouldn't be also capped up
to a io_combine_limit GUC (by default 16, so that's like 128kB per advise).
I mean, OSes should accept it, but based on my research they could silently
clamp it to sysfs per-device "read_ahead_kb" or 2MB (?), so if we would
merge here like way more and request e.g. 5MB in a single fadvise, it would
be simply ignored (??) and that would be a no-op, and after some time
fadvise performance effects would be lost.
So perhaps it's better to limit ourselves to that GUC (?) That would mean
we would somehow need to adjust the algorithm to fulfill the full window
of prefetches from the 2nd callsite too.

3. I would rename pf_index to just prefetch_idx, so it would easier to read.

-J.

Gustavo William

gustavowilliam0805@gmail.com

about 1 month ago

In reply to: Jakub Wartak (#3)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

Hi Jakub,

Thanks for reviewing it. I'm resending the chain of patches here, with 0005
already merged + your suggestions applied.

Here is a summary of what I've fixed:

I've forgotten to support incremental mode (which means that I ended up

issuing posix_fadvise SEQUENTIAL even in incremental mode

1. Fixed. Only issue posix_fadvise SEQUENTIAL for full backups and only
issue posix_fadvise WILLNEED for incremental backups.

there's bug that when m_io_c = 0 you still issue posix_fadvise()s from

2nd callsite (no way to turn it off) and also with e.g. m_io_c = 1 we seem
to issue an additional one.

2. Fixed this by only calling prefetch_next_incremental_run when m_io_c >
1. Also moved the call to happen after read_file_data_into_buffer instead
of before. Both changes together make the prefetching logic a little more
honest regarding I/O queue depth, i.e:

- When m_io_c = 0 or m_io_c = 1: no prefetching at all, queue depth is
constantly1 (the synchronous block read).

- When m_io_c > 1 (e.g. 2): prefetch 2 initially > read 1 + prefetch 1 >
read 1 + prefetch 1... on average, there will always be 2 I/O
simultaneously (assuming fadvise calls are honored).

I'm not sure f that merging of adjacent blocks shouldn't be also capped

up a io_combine_limit GUC

3. Agree that capping is good there. Though I think
maintenance_io_concurrency can be reused for this purpose instead? I think
a new GUC for this might be too specific. I currently have it implemented
using m_io_c. Let me know your thoughts on this if you think otherwise.

I would rename pf_index to just prefetch_idx, so it would easier to read.

4. Agreed. Fixed.

Thanks,
Gustavo Oliveira

Jakub Wartak

jakub.wartak@enterprisedb.com

27 days ago

In reply to: Gustavo William (#4)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On Tue, Sep 22, 2026 at 9:40 PM Gustavo William
<gustavowilliam0805@gmail.com> wrote:

Hi Jakub,

Thanks for reviewing it. I'm resending the chain of patches here, with 0005 already merged + your suggestions applied.

[..]

Hi Gustavo,

TLDR; I've verified ~9.4x improvement with 0005-incremental-backup just for
the read path on the NVMe (by avoiding stalled pread()). Good job :)

Testing method:
initdb # summarize_wal=on , maintenance_io_concurrency=2, ...
pgbench -i -s 500 --fillfactor=100 # ~7.5GB base/
pg_basebackup -D /tmp/full -c fast -v
pgbench -N -c 8 -j 8 -t 10000 postgres # -N=updates
psql -c "CHECKPOINT" postgres
sudo /usr/local/bin/drop_fs_cache.sh
time pg_basebackup -i /tmp/full/backup_manifest --target=server-blackhole \
-Xnone -c fast --no-sync --no-verify-checksums

Avg of 3 runs (drop_caches + time pg_basebackup -i)
Without 0005: 7.53s
With 0005: 0.80s (note it's just reading due to blackhole, so no writes!)

I don't have access to SATA drive, but I suspect it would be way better
there.

I was kind of sceptical of the result at first, but explanation of the
phehomena seems to be like this: basebackup_incremnetal.c has
GetFileBackupMethod()->qsort() so it sorts the blocks to be read. Without 0005
patch, preads() of increasing block numbers (offsets) all cause hit page cache
misses, and apparently - at least here on kernel 6.17.x, but earlier for sure
too - this seem to kick off the readahead (because kernel's readahead seems to
be implemented to only kick in by by page-cache misses , it won't even start
if there are hits, kind of makes sense), so pure preads() tend to overread a
lot (like reading N-times more data from a a 1GB segment than we need).

I think the kernel's assumption is that reading sequentially is cheaper than
reading one by one, but depending on density of changes we often - at least
in this cenario, that is in incremetanl basebackup mode - we read much less
than the kernel reads with readahead (simply said: it overreads a lot!).

So using fadvise(FADV_WILLNEED) here caused kind of disarms such aggressive
readahead completley (as it results in 100% hit to pagecaches, so readahead is
not even activated), and this causes way faster incremental times, so fast
that I'm had to actually reserach why and how it happens :o

Issues with the patch
- it seems maintenance_io_concurrency=1 disables prefetch? (it's not queue
depth, it's more of how many to prefetch, so "1" means IMHO pread() +
one posix_fadvise() or am I wrong?)
- pgindent might be needed.

-J.

Heikki Linnakangas

heikki.linnakangas@enterprisedb.com

20 days ago

In reply to: Jakub Wartak (#5)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

I started to look at these, starting from this patch:

0004 increases SINK_BUFFER_LENGTH from its current 32 kB. Profiling the
server side shows a huge number of small pread() calls coming from
basebackup_read_file(), which is surprising given that elsewhere in the tree
(backend/storage/buffer/README) we already mention that a 256 kB ring for seq.
scans scans is used because it fits comfortably in L2 cache. On my laptop,
with no SSL, no checksum generation or verification, a hot filesystem cache,
and a client blackhole to eliminate client I/O, a 10 GB backup over loopback
runs at:
3.1 GB/s with the 32 kB buffer,
5.1 GB/s at 128 kB,
5.5 GB/s at 256 kB,
6.0 GB/s at 1 MB
(those are average of five runs).

Cool

The win comes from letting pread()
swallow much larger chunks of each segment in one go: per-core L2 caches are
1-2 MB even on laptops these days, syscalls have become more expensive since
the Spectre/Meltdown mitigations, and the kernel's default readahead is
already in the 128-512 kB range, so there's little reason to trickle the data
through 32 kB at a time.

While we're at it, we should probably allocate the buffer with
palloc_aligned(PG_IO_ALIGN_SIZE); a pread() into an aligned buffer
should be a little faster.

I wonder about this in read_file_data_into_buffer():

cnt = basebackup_read_file(fd, sink->bbs_buffer,
Min(sink->bbs_buffer_length, length),
offset, readfilename, true);

/* Can't verify checksums if read length is not a multiple of BLCKSZ. */
if (!verify_checksum || (cnt % BLCKSZ) != 0)
return cnt;

It doesn't seem right to assume that the count must be a multiple of
BLCKSZ, especially if we use a larger buffer. Should we do something
about that?

- Heikki

Jakub Wartak

jakub.wartak@enterprisedb.com

19 days ago

In reply to: Heikki Linnakangas (#6)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On Mon, Oct 5, 2026 at 6:24 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

I started to look at these, starting from this patch:

Hi Heikki, thanks for taking a look !

0004 increases SINK_BUFFER_LENGTH from its current 32 kB. Profiling the
server side shows a huge number of small pread() calls coming from
basebackup_read_file(), which is surprising given that elsewhere in the tree
(backend/storage/buffer/README) we already mention that a 256 kB ring for seq.
scans scans is used because it fits comfortably in L2 cache. On my laptop,
with no SSL, no checksum generation or verification, a hot filesystem cache,
and a client blackhole to eliminate client I/O, a 10 GB backup over loopback
runs at:
3.1 GB/s with the 32 kB buffer,
5.1 GB/s at 128 kB,
5.5 GB/s at 256 kB,
6.0 GB/s at 1 MB
(those are average of five runs).

Cool

The win comes from letting pread()
swallow much larger chunks of each segment in one go: per-core L2 caches are
1-2 MB even on laptops these days, syscalls have become more expensive since
the Spectre/Meltdown mitigations, and the kernel's default readahead is
already in the 128-512 kB range, so there's little reason to trickle the data
through 32 kB at a time.

While we're at it, we should probably allocate the buffer with
palloc_aligned(PG_IO_ALIGN_SIZE); a pread() into an aligned buffer
should be a little faster.

I've attached patchset with 0004 modified that has this adjustement, but I
haven't quantified the gain for this alone though.

BTW: I've added discussion link and other tags, fixed some minor types in to all
commitmsgs too.

I wonder about this in read_file_data_into_buffer():

cnt = basebackup_read_file(fd, sink->bbs_buffer,
Min(sink->bbs_buffer_length, length),
offset, readfilename, true);

/* Can't verify checksums if read length is not a multiple of BLCKSZ. */
if (!verify_checksum || (cnt % BLCKSZ) != 0)
return cnt;

It doesn't seem right to assume that the count must be a multiple of
BLCKSZ, especially if we use a larger buffer. Should we do something
about that?

Your question has opened a whole can of worms on my side :(, but isn't this
code saying two things at once?
- it's only for calculating checksums only for BLCKSZ-sized blocks of
relations (I could imagine some user-placed file, or ENOSPC failure)
- we expect pread/pwrite() atomicity between two concurrent backends (one
backend doing pwrite()/fallocate() extends or truncation while the 2nd
is doing basebackup?)

bbsink_begin_backup(buffer_length=SINK_BUFFER_LENGTH) will set sink->
bbs_buffer_length to that SINK.. value and even there verify too:
Assert((sink->bbs_buffer_length % BLCKSZ) == 0);

sendFile() has the same (+ there's more occurences of this)
/*
* Checksums are verified in multiples of BLCKSZ, so the buffer length
* should be a multiple of the block size as well.
*/
Assert((sink->bbs_buffer_length % BLCKSZ) == 0);

further more sendFile() has this too above the "if (cnt < BLCKSZ)"
/*
* If we get a partial read, that must mean that the relation is
* being truncated. Ultimately, it should be truncated to a
* multiple of BLCKSZ, since this path should only be reached for
* relation files, but we might transiently observe an
* intermediate value.
*
* It should be fine to treat this just as if the entire block had
* been truncated away - i.e. fill this and all later blocks with
* zeroes. WAL replay will fix things up.
*/

and below that, there is ereport() that says just checksums couldn't be
verified. I mean in 0004 we are just growing and so it assumes we shouldn't
get any problems with this code. It looks like it is written while expecting
proper pread()/pwrite() atomicity when extending for up to 32kb, or am I
missing something?

On 6.17.x and 6.14.x x86_64 ext4 I've got no issues with such atomicity for
buffered preads while __appending__ stuff (using pwrite() to extend) and pread()
got correct stuff or premature EOF - that was for both 8kB and 256kB. Of
course I'm able to get torn-pages (only on 6.17.x - sic!) while doing buffered
OVERWRITES (so yea, 8kb split into 4kB + old 4KB -- checksum wouldn't match]).

Anyway as this topic touches the lack of pread()/pwrite() atomicity, but:
a) for torn-pages we are already protected there (see nearby comment for
reread_cnt , so we re-read)
b) for truncation too we are protected too
c) and it seems there is no issue for extending/appending
I've done some extensive pgbench read-write tests with pg_basebackup back then
and that such stuff didn't happen, perhaps buildfarm would tell more(?) My
only worry is that there could be some silent kernel change (in the past)
that would allow too see incompletle data by pread() for pwrite(), but if
that is not happening on 8kB/32kB (which is already over page-size), why
would that be happening for higher values?

BTW: previous such discussions [1]/messages/by-id/20220116071210.GA735692@rfd.leadboat.com[2]/messages/by-id/20220116210241.GC756210@rfd.leadboat.com[3]/messages/by-id/CA+hUKGJ98=MOjDCnMAC5gSpkzrrey=O+aEQJ1OY03C=cVtiwkA@mail.gmail.com[4]/messages/by-id/CA+hUKG+19bZKidSiWmMsDmgUVe=_rr0m57LfR+nAbWprVDd_cw@mail.gmail.com all seem to point that lack of
similar atomicity guaranteees or seem to be kernel bugs or some bizzare platform
(see e.g. "pwrite/pread is non-atomic" in [2]/messages/by-id/20220116210241.GC756210@rfd.leadboat.com) or at least deficit in fs
(while XFS being always safer [5]/messages/by-id/CA+hUKGJsT8G_YyjUzMZaJTWyua6PbwC3TAUMv_kDS0F0vzr2Pw@mail.gmail.com - after "As for ext4, we've detected and debugged clues..").

Still, even if if we just read miss some stale data due to pread() woes (and
skip checksums there) the data should be recoverable from WAL.

-J.

[1]: /messages/by-id/20220116071210.GA735692@rfd.leadboat.com
[2]: /messages/by-id/20220116210241.GC756210@rfd.leadboat.com
[3]: /messages/by-id/CA+hUKGJ98=MOjDCnMAC5gSpkzrrey=O+aEQJ1OY03C=cVtiwkA@mail.gmail.com
[4]: /messages/by-id/CA+hUKG+19bZKidSiWmMsDmgUVe=_rr0m57LfR+nAbWprVDd_cw@mail.gmail.com
[5]: /messages/by-id/CA+hUKGJsT8G_YyjUzMZaJTWyua6PbwC3TAUMv_kDS0F0vzr2Pw@mail.gmail.com - after "As for ext4, we've detected and debugged clues.."
- after "As for ext4, we've detected and debugged clues.."

Heikki Linnakangas

heikki.linnakangas@enterprisedb.com

19 days ago

In reply to: Jakub Wartak (#7)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On 06/10/2026 16:10, Jakub Wartak wrote:

further more sendFile() has this too above the "if (cnt < BLCKSZ)"
/*
* If we get a partial read, that must mean that the relation is
* being truncated. Ultimately, it should be truncated to a
* multiple of BLCKSZ, since this path should only be reached for
* relation files, but we might transiently observe an
* intermediate value.
*
* It should be fine to treat this just as if the entire block had
* been truncated away - i.e. fill this and all later blocks with
* zeroes. WAL replay will fix things up.
*/

and below that, there is ereport() that says just checksums couldn't be
verified. I mean in 0004 we are just growing and so it assumes we shouldn't
get any problems with this code. It looks like it is written while expecting
proper pread()/pwrite() atomicity when extending for up to 32kb, or am I
missing something?

I think that comment is wrong. A short read is possible for other
reasons than a truncated file, e.g. if the read is interrupted by a
signal. In the past, we've made that same assumption that you never get
a short read on a BLCKSZ-sized read in other places too, but it was
always questionable. Thomas Munro fixed it for smgrread() in commit
4908c58720.

I think we should now fix that in read_file_data_into_buffer(), too.
Make it retry, so that it never returns fewer bytes than requested
except for EOF. With a larger buffer, a short read becomes more likely.

- Heikki

Jakub Wartak

jakub.wartak@enterprisedb.com

18 days ago

In reply to: Heikki Linnakangas (#8)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On Tue, Oct 6, 2026 at 5:42 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

I think that comment is wrong. A short read is possible for other
reasons than a truncated file, e.g. if the read is interrupted by a
signal. In the past, we've made that same assumption that you never get
a short read on a BLCKSZ-sized read in other places too, but it was
always questionable. Thomas Munro fixed it for smgrread() in commit
4908c58720.

I think we should now fix that in read_file_data_into_buffer(), too.
Make it retry, so that it never returns fewer bytes than requested
except for EOF. With a larger buffer, a short read becomes more likely.

OK, I see, thanks, so I've tried this in the attached 0004.
0001..0004 without asserts, still give ~1.67x on hot DB fully in VFS cache:

pg_basebackup -c fast -v -h 127.0.0.1 --target=client-blackhole -Xnone \
-c fast --no-sync --no-verify-checksums --manifest-checksums=NONE \
2>&1 | grep MB/s
(client-blackhole still transfers to CLI, so it's receieving just ignoring
write path at all)

256kb (as within 0004):
pg_basebackup: base backup completed (avg 5503.2 MB/s)
pg_basebackup: base backup completed (avg 5331.5 MB/s)
pg_basebackup: base backup completed (avg 5205.6 MB/s)

32kB:
pg_basebackup: base backup completed (avg 3110.2 MB/s)
pg_basebackup: base backup completed (avg 3307.3 MB/s)
pg_basebackup: base backup completed (avg 3191.2 MB/s)

I've tried a couple of other integrity tests too for this new retry code:

1. high concurrency pgbench with loop of parallel pg_basebackup +
pg_verifybackup for couple of minutes (to see if that backing up
files being appended works OK). It went OK.

2. I've tried failure injection on 3rd invocation of pread of certain file:
strace -ff -p $POSTMASTERPID -P $PGDATA/base/1/1255 \
-e trace=pread64 -k -e inject=pread64:error=EINTR:when=3

and in first version of my patch I've got:
WARNING: aborting backup due to backend exiting before pg_backup_stop ..
ERROR: could not read file "./base/1/1255": Interrupted system call

strace got it exactly there:
pread64(14, ..., 262144, 524288) = -1 EINTR (Interrupted system
call) (INJECTED)

/usr/lib/x86_64-linux-gnu/libc.so.6(pread64+0x17) [0xfa577]
.../bin/postgres(basebackup_read_file+0x61) [0x23b9a1]
.../bin/postgres(read_file_data_into_buffer+0x4d) [0x23bcfd]

so right, today even master seems to be vulnerable to this I think.
Anyway, I've added to 0004 the EINTR guard (FileReadV() had goto for this)
and I have retested this using the EINTR injection for way more pgdata
files at specifc blocks and then checked if the resulting DB was fine:

rm -rf /tmp/full1
# ~10GB DB
pg_basebackup -c fast -v -h 127.0.0.1 -D /tmp/full1 -c fast
# strace on some relations with "when=1..2+1" has reported INJECTED
# messages (causing restart of the syscall in code)
pg_verifybackup /tmp/full1

and got "backup successfully verified"

3. With SIMULATE_SHORT_READ it also works OK.

-J.

#10

Heikki Linnakangas

heikki.linnakangas@enterprisedb.com

10 days ago

In reply to: Jakub Wartak (#9)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On 07/10/2026 12:55, Jakub Wartak wrote:

On Tue, Oct 6, 2026 at 5:42 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:

I think that comment is wrong. A short read is possible for other
reasons than a truncated file, e.g. if the read is interrupted by a
signal. In the past, we've made that same assumption that you never get
a short read on a BLCKSZ-sized read in other places too, but it was
always questionable. Thomas Munro fixed it for smgrread() in commit
4908c58720.

I think we should now fix that in read_file_data_into_buffer(), too.
Make it retry, so that it never returns fewer bytes than requested
except for EOF. With a larger buffer, a short read becomes more likely.

OK, I see, thanks, so I've tried this in the attached 0004.

Ok, looks good as far as it goes. But looking at basebackup_read_file(),
I think there's a race condition with partial_read_ok == false. This
could happen:

0. read_file_data_into_buffer() observes a checksum failure thanks to a
torn write, and retries reading a single BLCKSZ block.
1. basebackup_read_file() is called with partial_read_ok == false,
nbytes=BLCKSZ.
2. It reads the first 4096 bytes of the block. (e.g. because of
SIMULATE_SHORT_READ, or EINTR)
3. The file is truncated
4. basebackup_read_file() makes another pg_pread() call to read the rest
of the block. It returns 0, because the file was truncated
5. basebackup_read_file() throws a partial read error: "could not read
file \"%s\": read %zu of %zu".

If the truncation happened earlier, before the first read, or later,
after the whole block has been read, you don't get the error.

- Heikki

#11

Gustavo William

gustavowilliam0805@gmail.com

5 days ago

In reply to: Heikki Linnakangas (#10)

Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

Hi,

Apologies for the delay.

TL;DR:

- The speedup from 0005 depends a lot on how much the incremental backup
has changed
since the last one: ~9.8x at 14% delta, ~3.7x at 33%, ~1.2x at 57%. That
explains the
difference between Jakub's 9.4x speedup and the ~1.5x I first saw.
- Most of the gain comes from avoiding the kernel's own readahead on
scattered blocks
(it was doing more than double the I/O needed), not from concurrency.
- Merging adjacent blocks matters. Without it, 0005 is slower than master
on backups
with a high delta.
- maintenance_io_concurrency (m_io_c) beyond 2 only helps a little, and
only when
there are many adjacent blocks. m_io_c = 1 is still worse though.
- Code changes in this version: prefetch is enabled again for m_io_c = 1,
the merge
limit is now io_combine_limit, there is a new limit on how far ahead we
prefetch,
and the prefetch state moved into a new IncrPrefetch struct.

I've verified ~9.4x improvement with 0005-incremental-backup

I was confused by the big difference between our results, since I had
initially seen
only ~1.5x. After replicating your tests I found the reason: the speedup
varies a lot
depending on how much delta your incremental backup has.

I followed the same method you described:

initdb # summarize_wal=on, maintenance_io_concurrency=2, ...
pgbench -i -s 500 --fillfactor=100 -d dbname=test # ~7.5GB
pg_basebackup -D /pg-test/full -c fast
# ~14% of data change
psql test -c "CHECKPOINT"
./measure pg_basebackup --incremental /pg-test/full/backup_manifest
--wal-method none --target server-blackhole --checkpoint fast

# "measure" is a simple script that runs the command 3 times, dropping
caches in between

That gives an incremental backup of ~1.1G (~14% delta). To increase the
delta I then ran
pgbench + checkpoint a few more times between backups. Results (in seconds):

delta change without 0005 with 0005 speedup
------------- ------------ --------- -------
~14% (~1.1G) 42.48 4.31 ~9.8x
~33% (~2.5G) 27.56 7.49 ~3.7x
~57% (~4.3G) 15.60 13.13 ~1.2x

So the higher the delta, the smaller the gain from 0005. I think this ties
back to the
kernel behavior you observed.

Reading scattered blocks the way pg_basebackup does today seems to confuse
the kernel.
After a single block read, the kernel may readahead the following blocks,
expecting a
sequential pattern. With scattered delta blocks that never happens, so the
prefetched
data is wasted. I actually checked this on /proc/<pid>/io during a backup
of ~14% delta:

Without 0005
```
rchar: 1139995913 # bytes requested by the process (1.1GB)
syscr: 138285
read_bytes: 2557988864 # bytes actually read from storage (2.5GB)
```

With 0005
```
rchar: 1139966637 # bytes requested by the process (1.1GB)
syscr: 138279
read_bytes: 1143574528 # bytes actually read from storage (1.1GB)
```

So, without 0005 the kernel does more than double the I/O we need. 0005
helps with
concurrency, but its major improvement is avoiding this misguided readahead.

This explains the results above. With 14% delta from pgbench, most blocks
are
scattered, so 0005 avoids all that double I/O and gets a huge speedup. At
57% the delta
blocks are much more adjacent, so the kernel's own readahead gets a good
part of its
guesses right and 0005 has less to improve on.

A few other questions I tried to answer in this round:

1. Does increasing maintenance_io_concurrency make a big difference?

Not really. There's some gain up to a point, but it's small. This also
reinforces that
most of the gain of 0005 comes from avoiding the kernel's double I/O,
not from
concurrency itself. Below are some results of backup with 14% delta (in
seconds):

with m_io_c = 2: 4.45
with m_io_c = 16: 4.14
with m_io_c = 32: 4.07
with m_io_c = 64: 4.18

This made me wonder if we should stop making the prefetch window tied to
m_io_c and
instead just use a small fixed number like 2. But a higher m_io_c does
give a
modest gain on backups with a higher delta (more adjacent blocks). With
~57% delta:

with m_io_c = 2: 13.13
with m_io_c = 16: 10.28

2. Does maintenance_io_concurrency = 1 give any gain?

Yes, but less than 2. With ~14% delta (scattered blocks), in seconds:

with m_io_c = 2: 4.31
with m_io_c = 1: 6.94

3. Is merging adjacent blocks really helping?

At some point I considered dropping the merge-of-adjacent-blocks piece,
so I decided
to measure it on a 57% delta backup with m_io_c = 16 (results in
seconds):

without 0005: 15.60
0005 with merge: 10.28
0005 without merge: 20.41

It helps a lot. Without it, 0005 makes incremental backups with a high
delta
slower than not having 0005 at all (15.60 above).

My environment:

AWS EC2 c3.8xlarge; EBS vol 22000 IOPS, 1500MB/s max throughput;
RHEL 9.8; Kernel 5.14.0-687.41.1.el9_8.x86_64

Now regarding the code changes of this last iteration:

1. > it seems maintenance_io_concurrency=1 disables prefetch?

Yes, I had disabled it, thinking it wouldn't help. After the tests above
I enabled
it again.

2. We had previously agreed on a limit for how many blocks can be merged
into a single
fadvise. I was using maintenance_io_concurrency for that, but I thought
io_combine_limit might be a better fit according to its description. Do
you think
that makes sense?

3. The previous version had no limit on how far ahead, in blocks, we could
prefetch.
Each iteration runs one prefetch_next_incremental_run(), which can cover
many
blocks when they're adjacent. With lots of adjacent blocks we could get
far ahead of
the block being read, filling the page cache with pages that won't be
needed any
time soon and might even be evicted before they are read.

Based on that, I added a distance limit, in blocks, of
maintenance_io_concurrency * io_combine_limit. When there are many
adjacent blocks,
that keeps us at most maintenance_io_concurrency fadvise calls ahead of
the current
block.

4. I added a new struct named IncrPrefetch to the code to hold some
parameters of the
prefetch mechanism. I thought sendFile() was getting a lot of new
variables and
becoming harder to read. I hope that makes sense.

BTW thank you very much for helping with the reviews, I appreciate it.

Regards,
Gustavo Oliveira

Em qui., 15 de out. de 2026 às 09:08, Heikki Linnakangas <hlinnaka@iki.fi>
escreveu:

Show quoted text

On 07/10/2026 12:55, Jakub Wartak wrote:

On Tue, Oct 6, 2026 at 5:42 PM Heikki Linnakangas <hlinnaka@iki.fi>

wrote:

I think that comment is wrong. A short read is possible for other
reasons than a truncated file, e.g. if the read is interrupted by a
signal. In the past, we've made that same assumption that you never get
a short read on a BLCKSZ-sized read in other places too, but it was
always questionable. Thomas Munro fixed it for smgrread() in commit
4908c58720.

I think we should now fix that in read_file_data_into_buffer(), too.
Make it retry, so that it never returns fewer bytes than requested
except for EOF. With a larger buffer, a short read becomes more likely.

OK, I see, thanks, so I've tried this in the attached 0004.

Ok, looks good as far as it goes. But looking at basebackup_read_file(),
I think there's a race condition with partial_read_ok == false. This
could happen:

0. read_file_data_into_buffer() observes a checksum failure thanks to a
torn write, and retries reading a single BLCKSZ block.
1. basebackup_read_file() is called with partial_read_ok == false,
nbytes=BLCKSZ.
2. It reads the first 4096 bytes of the block. (e.g. because of
SIMULATE_SHORT_READ, or EINTR)
3. The file is truncated
4. basebackup_read_file() makes another pg_pread() call to read the rest
of the block. It returns 0, because the file was truncated
5. basebackup_read_file() throws a partial read error: "could not read
file \"%s\": read %zu of %zu".

If the truncation happened earlier, before the first read, or later,
after the whole block has been read, you don't get the error.

- Heikki

enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

Attachments:

Attachments:

Attachments:

Attachments:

Attachments:

Attachments: