[PG19] Promotion hangs behind a backend blocked on a stuck client

Started by shihao zhong10 days ago41 messageshackers
Beta feature

Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.

appliessuccessCI history

You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:

docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254201
psql -h localhost -U postgres

Built from patchset v40 (message #40), October 25, 2026 at 12:35 AM.

Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:

git clone --branch t254201_40 https://github.com/hackorum-dev/postgres.git

In a checkout you already have, add the fork once:

git remote add hackorum https://github.com/hackorum-dev/postgres.git

then, for this patchset and every later one:

git fetch hackorum t254201_40 && git checkout t254201_40

Patchset v40 (message #40) is on t254201_40

Jump to latest
#1shihao zhong
zhong950419@gmail.com

Hi,

Since 67c20979ce7, promotion waits until every process absorbs a
ProcSignalBarrier, even with wal_level=replica and no slots. A backend
blocked sending to a client never absorbs it. If a client on the standby
pauses while fetching a result, promotion has no upper bound. A paused
client that stayed connected still held it after 300 seconds. If the
client goes away, promotion waits until TCP gives up, about 16 minutes.
18.6 promotes in 0.1 seconds. This is item 8 in [1]/messages/by-id/CAGRkXqSFdjqZP_kwRz7dv6LHX-CSrkDnynNXgntRsCEnQTKxmg@mail.gmail.com.

Manu found why in [2]/messages/by-id/179191664881.1698521.9947772831570083261@gmail.com. ProcessClientWriteInterrupt() only acts on
ProcDiePending, so the barrier stays pending while secure_write() waits.
Once the send buffer is full it hangs every time.

The same wait blocks crash failover and standby
replay when the primary creates its first logical slot. With
synchronous_commit=remote_apply, commits on the primary then wait too.
DROP DATABASE has had the same problem since 15.

The patch absorbs barriers after a blocked wait in secure_read() or
secure_write() wakes up, if interrupts are not held. The normal path
does not change. We may be in the middle of a message, and
AbsorbDataChecksumsBarrier() can throw. So any error is made FATAL and
kept from the client, as Andres suggested in [3]/messages/by-id/CAO6_XqoZb2czT=i=VBQjwBwwo3d21cnMm-EpfYfx0m-O+abFrQ@mail.gmail.com. With a temporary
injection point there, the client sees the connection close. Without
that, the error lands inside a DataRow and the client hangs.

On the REL_19_STABLE head 84572589029 on Linux, all 38 runs with a stuck
client hung on 19, and all finished within 210 ms with the patch. The
attached file has each case and the controls.

DROP DATABASE waits for a barrier too. A stuck client has blocked
it since 15, and 14 is fine. 0001 applies to 15 and later as is, so I
think it should go back to 15.

Not covered is a backend blocked while sending an error message, where
interrupts are held. Andres described it in [3]/messages/by-id/CAO6_XqoZb2czT=i=VBQjwBwwo3d21cnMm-EpfYfx0m-O+abFrQ@mail.gmail.com.

[1]: /messages/by-id/CAGRkXqSFdjqZP_kwRz7dv6LHX-CSrkDnynNXgntRsCEnQTKxmg@mail.gmail.com
/messages/by-id/CAGRkXqSFdjqZP_kwRz7dv6LHX-CSrkDnynNXgntRsCEnQTKxmg@mail.gmail.com
[2]: /messages/by-id/179191664881.1698521.9947772831570083261@gmail.com
[3]: /messages/by-id/CAO6_XqoZb2czT=i=VBQjwBwwo3d21cnMm-EpfYfx0m-O+abFrQ@mail.gmail.com
/messages/by-id/CAO6_XqoZb2czT=i=VBQjwBwwo3d21cnMm-EpfYfx0m-O+abFrQ@mail.gmail.com

Attachments:

t254201_1
v2-0001-Absorb-ProcSignalBarriers-while-waiting-for-the-c.patchapplication/octet-stream; name=v2-0001-Absorb-ProcSignalBarriers-while-waiting-for-the-c.patchDownload+32-1
v2-0002-Test-barrier-absorption-with-a-stuck-client.patchapplication/octet-stream; name=v2-0002-Test-barrier-absorption-with-a-stuck-client.patchDownload+81-1
#2Fujii Masao
masao.fujii@gmail.com
In reply to: shihao zhong (#1)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Thu, Oct 15, 2026 at 12:54 PM shihao zhong <zhong950419@gmail.com> wrote:

Hi,

Since 67c20979ce7, promotion waits until every process absorbs a
ProcSignalBarrier, even with wal_level=replica and no slots.

Promotion currently waits for the barrier even with wal_level = logical?
Is that really necessary? XLogLogicalInfoActive() seems to always return
true in that case, regardless of the cached value.

Even with your patch allowing backends to process barriers while blocked
on client I/O, in addition to that fix, it's also worth skipping this barrier
when wal_level = logical? That would avoid unnecessary synchronization
with all running processes and could reduce promotion time.

Regards,

--
Fujii Masao

Attachments:

nocfbot-WIP-v1-0001-Skip-the-end-of-recovery-logical-WAL-barrier-at-w.patchapplication/octet-stream; name=nocfbot-WIP-v1-0001-Skip-the-end-of-recovery-logical-WAL-barrier-at-w.patchDownload+6-4
#3Manu
manuelreyesbravo@gmail.com
In reply to: shihao zhong (#1)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Fujii, Shihao,

Thanks for turning the diagnosis into a patch, Shihao. Fujii, good
question on whether the barrier is needed at all; I went and measured it.

it's also worth skipping this barrier when wal_level = logical?

Yes, and a bit more. Your WIP skips it for wal_level=logical and that
case promotes (the base hangs there 6/6). But the reported case is
wal_level=replica with no slots, and there the barrier is still emitted
and still hangs (12/12 here).

The wait isn't needed when the new status is "off" either: a process
still caching "on" only writes extra logical info, which is safe, the
same reasoning DisableLogicalDecoding() already uses to emit without
waiting. So the barrier only needs waiting on when promotion turns the
status ON. The attached 0003 (on top of Shihao's 0001) does that:
replica and logical both promote, including the mid-error case v2 can't
reach; logical decoding stays correct (48/48, calibrated against a
no-barrier mutant); the recovery suite and make check pass; and healthy
promotion is unchanged (~5 ms).

It doesn't replace Shihao's patch. When the standby already has logical
slots the wait is real, and there 0003 hangs while v2 fixes it. The two
are complementary.

One note on the v2 test: without the fix the promotion subtest still
passes most of the time (the send buffer has to fill first), so it
doesn't reliably catch that case; the DROP DATABASE subtest fails every
time. On 15 and 16 the test exits early since psql there uses
FETCH_COUNT cursors, and the meson.build hunk doesn't apply on the
stable branches. 0001 itself applies cleanly from 14 to master.

Full measurements attached.

Regards,
Manu

Attachments:

t254201_3
0003-wait-end-of-recovery-barrier-only-when-enabling.patchtext/x-patchDownload+17-3
evidence-for-list.txttext/plainDownload
#4Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: shihao zhong (#1)
RE: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Shihao,

While discussing with LLM, I found that the current code might stuck in future,
when the callbacks in ProcessProcSignalBarrier() returns false.

According to the code comment, it can return false if it's not an appropriate
time thus we must try once again. At that time, the latch seems not to be set.
I came up with the below case for read:

1. callbacks in ProcessProcSignalBarrier() returns false.
2. it returns without seting the latch again.
3. secure_read() tries to read from the socket, but nothing found again
(errno == EAGAIN)
4. The backend reaches WaitEventSetWait() again. The caller side of the barrier
also keeps waiting.

I could reproduce with your patch set and below change. The reproducer could
stuck again on my env.

```
 bool
 ProcessBarrierSmgrRelease(void)
 {
+       static bool defer_once = true;
+
+       /*
+        * If this called from ProcessClientBarrierInterrupt() for the first time,
+        * do no-op and returns false. This simulates the case we have to try once
+        * again.
+        */
+       if (ExitOnAnyError && defer_once)
+       {
+               defer_once = false;
+               return false;
+       }
+
```

Actually, I'm quite unclear how we should fix. Should we just note the risk
somewhere because it does not happen yet? Another primitive idea is to set the
latch again in ProcessClientBarrierInterrupt() if ProcSignalBarrierPending is still
true, but it may waste the CPU time, wait in the secure_read() may finish soon
and check the socket again may times.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

#5Fujii Masao
masao.fujii@gmail.com
In reply to: Manu (#3)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Thu, Oct 15, 2026 at 3:51 PM Manu <manuelreyesbravo@gmail.com> wrote:

The wait isn't needed when the new status is "off" either: a process
still caching "on" only writes extra logical info, which is safe, the
same reasoning DisableLogicalDecoding() already uses to emit without
waiting.

0003 looks good to me. I've attached an updated version with revised
comments in UpdateLogicalDecodingStatusEndOfRecovery(), which still
claimed that all processes had updated their XLogLogicalInfo state
before the function returned.

I think it's better to backpatch this to v19, since commit 67c20979ce7
introduced the unnecessary barrier wait. Thoughts?

Barring any objections, I'm thinking to commit it.

Regards,

--
Fujii Masao

Attachments:

nocfbot-v3-0001-Avoid-unnecessary-end-of-recovery-barrier-waits.patchapplication/octet-stream; name=nocfbot-v3-0001-Avoid-unnecessary-end-of-recovery-barrier-waits.patchDownload+21-13
#6Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Fujii Masao (#5)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Thu, Oct 15, 2026 at 9:56 AM Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Oct 15, 2026 at 3:51 PM Manu <manuelreyesbravo@gmail.com> wrote:

The wait isn't needed when the new status is "off" either: a process
still caching "on" only writes extra logical info, which is safe, the
same reasoning DisableLogicalDecoding() already uses to emit without
waiting.

0003 looks good to me. I've attached an updated version with revised
comments in UpdateLogicalDecodingStatusEndOfRecovery(), which still
claimed that all processes had updated their XLogLogicalInfo state
before the function returned.

I think it's better to backpatch this to v19, since commit 67c20979ce7
introduced the unnecessary barrier wait. Thoughts?

Barring any objections, I'm thinking to commit it.

Thank you for the patch. It looks good to me, and I agree that it
should be backpatched to v19.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#7Manu
manuelreyesbravo@gmail.com
In reply to: Fujii Masao (#5)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Fujii,

I think it's better to backpatch this to v19, since commit 67c20979ce7
introduced the unnecessary barrier wait. Thoughts?

+1, and thanks for the comment fixes. Only the comments changed from
0003, and they read right to me. I ran v3 on REL_19_STABLE: it applies
cleanly there and on master, and make check and the recovery suite pass.
Unpatched 19 hung 12/12 on promotion with one backend stuck writing to a
client; with v3 it promoted 12/12 in 10-27 ms. Unpatched 18 promoted
12/12 in the same time, so 19 is the only branch that needs it.

v3 only covers promotion without logical slots. Shihao's 0001, which
came with his report, is the general fix for when the wait is real.
Since 0001 and his test will need backpatching too, I ran them on 15
through master. Shihao, the full numbers are in the attachment; in
short:

- 060 is now taken on every branch (060_multixact_crashed_updater.pl
went in today), so the test needs another number. 061 is free
everywhere. Renumbered and otherwise unchanged, it passes with 0001
on 17 through master, recovery suite included.
- On 15 and 16 it dies before testing anything: psql there runs
FETCH_COUNT through a cursor, so the backend never blocks. psql()
there also returns 0 on timeout, so the DROP DATABASE check can pass
while DROP DATABASE is still stuck.
- Once v3 is in, promotion with no slot doesn't wait at all, so the
promotion subtest passes without 0001.

The attached diff is a suggestion for all three: a COPY TO STDOUT writer,
a timed_out check, and a logical slot on the standby (19 and master
only). With it the test fails without 0001 and passes with it on 15
through master. The COPY writer also helps on its own: on unpatched
master the promotion subtest caught the hang 5/5 with it, against 1/5
as it stands. It's your test, so take whatever is useful.

Hayato, I could reproduce what you found: with your change, DROP DATABASE
stays blocked behind a backend in COPY FROM STDIN, 6/6, with or without
v3. So it's in 0001's absorb path, and v3 neither causes nor fixes it.
It's latent today: every barrier callback, in 19 and master, returns
true or throws ERROR, and none of about 1800 calls over the recovery
suite returned false. If 0001 wants to guard against it, setting the
latch again when ProcSignalBarrierPending is still set (second attached
diff) cleared your reproducer 3/3. A plain Assert there could fire
spuriously, since a new barrier can arrive right after the call.

Regards,
Manu

Attachments:

evidence-for-list.txttext/plainDownload
nocfbot-0002-test-suggestions.difftext/x-patchDownload+16-7
nocfbot-0001-setlatch-suggestion.difftext/x-patchDownload+7-0
#8shihao zhong
zhong950419@gmail.com
In reply to: Manu (#7)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi,

Hayato, you are right, thanks. v4-0001 sets the latch again if a
barrier is still pending, as Manu suggested. With your change on top,
DROP DATABASE waited 60 seconds with v2-0001 and 15 ms with v4-0001.

Manu, I took all of your test changes. On 19 and master the test fails
without 0001 and passes with it.

I call these v4 so they don't get mixed up with Fujii's v3.

Thanks,
Shihao

Attachments:

t254201_8
v4-0002-Test-barrier-absorption-with-a-stuck-client.patchapplication/x-patch; name=v4-0002-Test-barrier-absorption-with-a-stuck-client.patchDownload+90-1
v4-0001-Absorb-ProcSignalBarriers-while-waiting-for-the-c.patchapplication/x-patch; name=v4-0001-Absorb-ProcSignalBarriers-while-waiting-for-the-c.patchDownload+41-1
#9Manu
manuelreyesbravo@gmail.com
In reply to: shihao zhong (#8)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

DROP DATABASE waited 60 seconds with v2-0001 and 15 ms with v4-0001.

Thanks, and nicely done pulling it together -- 60 s to 15 ms is a good
result. The test changes read right to me too, and the regression is
pinned now: it fails without 0001 and passes with it on 19 and master.

It's been a pleasure to help on this one. I'll keep an eye on the thread
and jump in wherever it is useful, the backpatch included -- the test
will want renumbering there, since 060 is taken on every branch now.

Regards,
Manu

#10Fujii Masao
masao.fujii@gmail.com
In reply to: Masahiko Sawada (#6)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Fri, Oct 16, 2026 at 2:40 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

Thank you for the patch. It looks good to me, and I agree that it
should be backpatched to v19.

I've pushed the patch. Thanks!

Regards,

--
Fujii Masao

#11Manu
manuelreyesbravo@gmail.com
In reply to: Fujii Masao (#10)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Fujii,

I've pushed the patch. Thanks!

Thanks for taking it through, and for the comment pass on v3. Glad it
made 19.

Regards,
Manu

#12Masahiko Sawada
sawada.mshk@gmail.com
In reply to: shihao zhong (#8)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Thu, Oct 15, 2026 at 8:27 PM shihao zhong <zhong950419@gmail.com> wrote:

Hi,

Hayato, you are right, thanks. v4-0001 sets the latch again if a
barrier is still pending, as Manu suggested. With your change on top,
DROP DATABASE waited 60 seconds with v2-0001 and 15 ms with v4-0001.

Manu, I took all of your test changes. On 19 and master the test fails
without 0001 and passes with it.

I agree that 0001 is what fixes the rest. I'm reviewing it now, but
what I'm unsure about is whether to include it in v19 at this point.
It creates a new place where barrier callbacks run in the middle of a
protocol message, with any ERROR upgraded to FATAL, and every barrier
type we add later will have to be safe there. Neither of the two types
in v19 throws today, so this is more about what could break there
later, but it's the kind of change I'd want more time on than RC gives
us.

From the numbers upthread, a backend blocked writing to the client
(for example COPY TO STDOUT, or taking a base backup) looks like the
only case that is both likely and unbounded; the others either release
themselves or can't happen on a standby. It also needs a consumer that
has stopped rather than one that is just slow.

There are workarounds in the meantime. Setting wal_level=logical on
the standby skips the barrier entirely thanks to the recent commit. If
it does happen, the startup process logs the holding PID every five
seconds, and pg_terminate_backend() on that PID lets the promotion
finish.

So I'm currently leaning towards master only and we live with the
restriction in v19. Feedback is very welcome.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#13Manu
manuelreyesbravo@gmail.com
In reply to: Masahiko Sawada (#12)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

So I'm currently leaning towards master only and we live with the
restriction in v19. Feedback is very welcome.

+1 to master only. I tried to break the three assumptions on current
REL_19_STABLE (c705304a0f4, which has 239c4d30699), one fresh
primary/standby pair per run, 97 runs, and they hold up:

- Since 239c4d30699 the barrier wait only happens on a wal_level=replica
standby that has a logical slot on it -- one created on the standby, or
a failover slot synced by the slotsync worker (I measured both). Without
such a slot a stuck COPY TO STDOUT or a stuck base backup no longer
delays promotion (promoted 6/6, 10-27 ms). So the restriction is a bit
narrower than it first looks.

- With the slot, the only unbounded holders were backends blocked writing
to a client that stopped reading: COPY TO STDOUT, a large SELECT, and a
base backup (still waiting past 300 s). Idle, idle in transaction,
CPU-bound queries, pg_sleep, advisory-lock and AccessExclusiveLock waits
(including a lock the startup process itself holds), slow-but-alive
readers at 256 KB/s, and cascading/logical walsenders all promoted in
under 80 ms. A client stuck in authentication releases itself at
authentication_timeout.

- Both workarounds held: wal_level=logical on the standby promoted 19/19
in ~20 ms with the same stuck clients, and pg_terminate_backend() on the
PID the startup process logs every 5 s finished the promotion in 9-73 ms
in every hung run.

One edge worth a footnote in the docs or release notes: if the stuck write
is an ErrorResponse rather than a data message, pg_terminate_backend() does
not release it, because PostgresMain sends the error with interrupts held
(HOLD_INTERRUPTS before EmitErrorReport). That is not new in 19 and the
trigger I used to reach it is contrived, but there the ways out are killing
the client connection or tcp_user_timeout on the standby (10 s -> promoted
in ~7 s); wal_level=logical avoids it too.

Happy to share the scripts and the raw per-run numbers if they're useful.

Regards,
Manu

#14shihao zhong
zhong950419@gmail.com
In reply to: Masahiko Sawada (#12)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Masahiko,

I would still like 0001 in 19, because the cases that remain are the
HA ones.

After 239c4d30699 the wait only happens when the standby has a logical
slot, and Manu measured that a failover slot synced by the slotsync
worker is enough. That is exactly the setup of an HA cluster that
keeps logical replication across failover.

A network split or a dead app host leaves such a client behind.
Promotion then waits until TCP gives up, which may took up to 15 minutes

The workarounds need someone to know about this ahead of time, and
most failover tools do not terminate backends before promoting.

If you still prefer master only, I am fine with it and we need to
add that into the doc note for 19.

Thanks,
Shihao

#15Masahiko Sawada
sawada.mshk@gmail.com
In reply to: shihao zhong (#14)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Sun, Oct 18, 2026 at 5:14 PM shihao zhong <zhong950419@gmail.com> wrote:

Hi Masahiko,

I would still like 0001 in 19, because the cases that remain are the
HA ones.

After 239c4d30699 the wait only happens when the standby has a logical
slot, and Manu measured that a failover slot synced by the slotsync
worker is enough. That is exactly the setup of an HA cluster that
keeps logical replication across failover.

A network split or a dead app host leaves such a client behind.
Promotion then waits until TCP gives up, which may took up to 15 minutes

The workarounds need someone to know about this ahead of time, and
most failover tools do not terminate backends before promoting.

If you still prefer master only, I am fine with it and we need to
add that into the doc note for 19.

TBH I'm still unsure that the 0001 patch is the correct approach.
ProcessProcSignalBarrier() is written on the assumption that a handler
can raise an error. That's why it has a PG_TRY block for a later
retry. In v19 we have only two handlers, ProcessBarrierSmgrRelease()
and ProcessBarrierUpdateXLogLogicalInfo(), and neither raises an
error. But on master, AbsorbDataChecksumsBarrier() raises an error on
purpose, with a comment saying it "will be caught by the procsignal
machinery for a later retry". With 0001 that error would terminate the
session instead of being retried. So the assumption that 0001 adds is
already false on master.

Other possible solutions are: (1) send the signal barrier at promotion
only when we really need to send it. In other words, if we know that
logical decoding has been enabled from the startup and no change
happened during the recovery, we can skip it even if wal_level =
replica. (2) the startup process terminates the backend who doesn't
absorb the barrier, like as we resolve the recovery conflicts.

(1) would narrow further the case where we need the wait, but doesn't
eliminate the case completely. (2) sounds more promising to me, and we
already have similar machinery such as
ResolveRecoveryConflictWithVirtualXIDs(). Probably it could help DROP
DATABASE too: TerminateOtherDBBackends() terminates only the backends
connected to the target database, so FORCE doesn't help today if a
backend connected to another database doesn't absorb the SMGRRELEASE
barrier.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

#16shihao zhong
zhong950419@gmail.com
In reply to: Masahiko Sawada (#15)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Masahiko,

I think you are right, 0001 would turn the checksum barrier's retry into a
FATAL.

Option (1) can cover the failover slot case in 19. At the end of
recovery, a stale cache can only be in a process started before
StartupLogicalDecodingStatus(), like the checkpointer. Later ones read
it at startup, and a replayed enable waits for everyone. So the startup
process can emit the barrier there, and at the end of recovery wait for
that generation when the status did not change.

It does not help when replaying the enable record has to wait, or
when the end of recovery itself turns the status on. For master I
agree (2) is better, and it helps DROP DATABASE too.

Thanks,
Shihao

Attachments:

t254201_16
v5-0002-Test-promotion-with-a-backend-blocked-on-a-stuck-.patchapplication/octet-stream; name=v5-0002-Test-promotion-with-a-backend-blocked-on-a-stuck-.patchDownload+124-1
v5-0001-Skip-the-end-of-recovery-barrier-wait-when-nothin.patchapplication/octet-stream; name=v5-0001-Skip-the-end-of-recovery-barrier-wait-when-nothin.patchDownload+35-5
nocfbot-v5-19-0001-Skip-the-end-of-recovery-barrier-wait-when-nothin.patchapplication/octet-stream; name=nocfbot-v5-19-0001-Skip-the-end-of-recovery-barrier-wait-when-nothin.patchDownload+159-5
#17Manu
manuelreyesbravo@gmail.com
In reply to: shihao zhong (#16)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Shihao,

At the end of recovery, a stale cache can only be in a process started
before StartupLogicalDecodingStatus(), like the checkpointer. Later ones
read it at startup, and a replayed enable waits for everyone.

I checked that directly for the failover-slot case (slot present before the
base backup, synced by slotsync, no status change replayed): with gdb I read
the process-local XLogLogicalInfo of every standby child just before
pg_promote(), over 8 runs. The only stale ones were the checkpointer, the
background writer, the two io workers and the startup process; every client
backend (including the one blocking promotion), the walreceiver and the
slotsync worker already had it set. So at end of recovery the barrier only
has to reach those auxiliaries, and the backend that blocks promotion is
already up to date. (I did not check whether any of those auxiliaries writes
WAL that needs the updated status.)

It does not help when replaying the enable record has to wait, or
when the end of recovery itself turns the status on. For master I
agree (2) is better, and it helps DROP DATABASE too.

To check that (2) really clears these cases, I wrote a small,
measurement-only prototype of Sawada-san's idea on master (0497318e442):
in WaitForProcSignalBarrier(), after a fixed grace, a client backend or
walsender that still has not absorbed the barrier gets SIGTERM, as
pg_terminate_backend() does. It is not a patch proposal; the direction is
Sawada-san's and yours, and the diff is yours to use or ignore. (I built
your v5-0001 on the same base to compare the numbers below.)

Each case had one backend stuck in COPY TO STDOUT to a client that
stopped reading, a 2 s grace, and 3 runs:

- promotion, logical decoding enabled since startup: master hung, the
prototype promoted in ~2.02 s, v5 in 7-22 ms
- promotion after an enable record was replayed: the same
- replay of the enable record with the backend already stuck: master
and v5 stayed stopped for the whole 15 s, the prototype caught up in
~2.03 s
- pg_enable_data_checksums() on a primary: master was still
"inprogress-on" after 15 s, the prototype reached "on" in ~2.1 s
- DROP DATABASE with the backend connected to another database: master
hung for 20 s, the prototype dropped it in ~2.01 s

The stuck backend always ended with "terminating connection due to
administrator command". With a 5 s grace the times became ~5.0-5.1 s,
and the termination never came before the grace. As a negative control
I ran five idle sessions, one in pg_sleep() and one slow reader
(256 KB/s) through promotion, checksum enabling and DROP DATABASE: all
9 runs finished in 10-111 ms, and none of the 63 sessions was
terminated.

Two findings that may matter for the design:

- The recovery-conflict signal, as it is, does not reach such a backend.
The same prototype sending SignalRecoveryConflict() instead of SIGTERM
released none of them (0/12 across promotion, replay, checksums and
DROP DATABASE), since ProcessClientWriteInterrupt() acts only on
ProcDiePending. The same is true today for real conflicts: with
max_standby_streaming_delay = 1s, a snapshot conflict against that
backend kept replay stopped for the whole 15 s (3/3); the startup
process named it as the conflicting process, and confl_snapshot
stayed 0.

- For DROP DATABASE, a stuck backend connected to the target database
is not a barrier wait: a plain DROP fails after 5 s with "is being
accessed by other users", and WITH (FORCE) already works on master
(~115 ms). The barrier case is a backend on another database, as
Sawada-san said.

I can share the scripts, the diff and the per-run numbers if that is useful.

Regards,
Manu

#18Amit Kapila
amit.kapila16@gmail.com
In reply to: Masahiko Sawada (#15)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Tue, Oct 20, 2026 at 2:47 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Sun, Oct 18, 2026 at 5:14 PM shihao zhong <zhong950419@gmail.com> wrote:

TBH I'm still unsure that the 0001 patch is the correct approach.
ProcessProcSignalBarrier() is written on the assumption that a handler
can raise an error. That's why it has a PG_TRY block for a later
retry. In v19 we have only two handlers, ProcessBarrierSmgrRelease()
and ProcessBarrierUpdateXLogLogicalInfo(), and neither raises an
error. But on master, AbsorbDataChecksumsBarrier() raises an error on
purpose, with a comment saying it "will be caught by the procsignal
machinery for a later retry". With 0001 that error would terminate the
session instead of being retried. So the assumption that 0001 adds is
already false on master.

Other possible solutions are: (1) send the signal barrier at promotion
only when we really need to send it. In other words, if we know that
logical decoding has been enabled from the startup and no change
happened during the recovery, we can skip it even if wal_level =
replica. (2) the startup process terminates the backend who doesn't
absorb the barrier, like as we resolve the recovery conflicts.

(1) would narrow further the case where we need the wait, but doesn't
eliminate the case completely.

Right, for example, If the first logical slot appeared on the primary
shortly before promotion, replay toggled the status (false→true), the
case counts as "changed," and the wait remains. So, the same problem
still exists.

(2) sounds more promising to me, and we
already have similar machinery such as
ResolveRecoveryConflictWithVirtualXIDs().

Like ResolveRecoveryConflictWithVirtualXIDs(), we may want to wait for
sometime (which needs GUC or a fixed time wait) before terminating the
backend. I think this is a direction worth pursuing but I feel it is
too late for v19 considering the amount/complexity of changes
required. I feel for v19, we can add a Note in the docs with a
workaround and then discuss the more robust solution for HEAD.

--
With Regards,
Amit Kapila.

#19Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Amit Kapila (#18)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

On Tue, Oct 20, 2026 at 12:20 AM Amit Kapila <amit.kapila16@gmail.com> wrote:

On Tue, Oct 20, 2026 at 2:47 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:

On Sun, Oct 18, 2026 at 5:14 PM shihao zhong <zhong950419@gmail.com> wrote:

TBH I'm still unsure that the 0001 patch is the correct approach.
ProcessProcSignalBarrier() is written on the assumption that a handler
can raise an error. That's why it has a PG_TRY block for a later
retry. In v19 we have only two handlers, ProcessBarrierSmgrRelease()
and ProcessBarrierUpdateXLogLogicalInfo(), and neither raises an
error. But on master, AbsorbDataChecksumsBarrier() raises an error on
purpose, with a comment saying it "will be caught by the procsignal
machinery for a later retry". With 0001 that error would terminate the
session instead of being retried. So the assumption that 0001 adds is
already false on master.

Other possible solutions are: (1) send the signal barrier at promotion
only when we really need to send it. In other words, if we know that
logical decoding has been enabled from the startup and no change
happened during the recovery, we can skip it even if wal_level =
replica. (2) the startup process terminates the backend who doesn't
absorb the barrier, like as we resolve the recovery conflicts.

(1) would narrow further the case where we need the wait, but doesn't
eliminate the case completely.

Right, for example, If the first logical slot appeared on the primary
shortly before promotion, replay toggled the status (false→true), the
case counts as "changed," and the wait remains. So, the same problem
still exists.

(2) sounds more promising to me, and we
already have similar machinery such as
ResolveRecoveryConflictWithVirtualXIDs().

Like ResolveRecoveryConflictWithVirtualXIDs(), we may want to wait for
sometime (which needs GUC or a fixed time wait) before terminating the
backend. I think this is a direction worth pursuing but I feel it is
too late for v19 considering the amount/complexity of changes
required. I feel for v19, we can add a Note in the docs with a
workaround and then discuss the more robust solution for HEAD.

Thank you for the comments.

shihao zhong drafted (1) the idea above upthread[1]/messages/by-id/CAGRkXqR8Kfdd=G1Le=RzhRHzZBbLtPc44207bwhrPDu_C5ZMEA@mail.gmail.com. There seems to be
room for improvement there and the patch would look like the attached:
it initializes the status in shared memory from the control file, so
that the processes the postmaster forks before the startup process
reads the checkpoint record don't start with a stale cache. Given the
patch and analysis[2]/messages/by-id/179247119467.2410654.17218914322784616346@gmail.com shared by Manu, let me summarize the issue and
solutions.

With the v6 patch, we would need to wait for the signal barrier to be
absorbed only when the startup process enables logical decoding at the
end of recovery, which is a rare case. On standbys the status is
toggled by replaying an XLOG_LOGICAL_DECODING_STATUS_CHANGE record,
and the enable case waits for all processes. In some corner cases, the
startup process enables logical decoding at the end of recovery on
behalf of a backend creating a logical slot, so we need a wait for
that case. While making this case rarer is appealing, the v6 patch
doesn't remove it.

The v6 patch also doesn't resolve the case where the replay itself
stops because a backend is stuck writing to a client. As Amit
mentioned, if that delay happens shortly before the promotion, the
promotion would get blocked. The same can be applied to recovery
conflict cases: Manu measured that such a backend is not released by a
recovery conflict signal either[2]/messages/by-id/179247119467.2410654.17218914322784616346@gmail.com, since
ProcessClientWriteInterrupt() services only ProcDiePending, and
therefore the recovery gets blocked even with
max_standby_streaming_delay set.

So I think that the recovery (and the promotion) getting stuck due to
client write I/O is not new in v19. max_standby_streaming_delay
doesn't work against a backend stuck writing to a client, and v19 just
adds one more way to hit the same limitation. I still prefer dealing
with this issue only for master, and leave the code in v19 as it is.
For v19 I think we should add a note to the documentation, as Amit
suggested. And possibly we can backpatch the v6 patch in a later minor
release once we test the approach enough on the master.

Regards,

[1]: /messages/by-id/CAGRkXqR8Kfdd=G1Le=RzhRHzZBbLtPc44207bwhrPDu_C5ZMEA@mail.gmail.com
[2]: /messages/by-id/179247119467.2410654.17218914322784616346@gmail.com

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

t254201_19
v6_fix_masahiko.patchtext/x-patch; charset=US-ASCII; name=v6_fix_masahiko.patchDownload+66-12
#20shihao zhong
zhong950419@gmail.com
In reply to: Masahiko Sawada (#19)
Re: [PG19] Promotion hangs behind a backend blocked on a stuck client

Hi Masahiko,

I agree. 19 gets a note in the docs, and v6 goes to master first.

The docs patch for 19 is attached. It says what blocks, how to see the
PID, and how to get out. I am happy to change the wording.

For (2), I think it is still worth doing on master even with v6. It
would also cover replay of the enable record and recovery conflicts
with a backend stuck on a client write. Manu, do you want to post it?
If not, I am happy to take it.

Thanks,
Shihao

Attachments:

nocfbot-v1-19-0001-doc-Note-that-promotion-can-wait-for-a-backend-st.patchapplication/octet-stream; name=nocfbot-v1-19-0001-doc-Note-that-promotion-can-wait-for-a-backend-st.patchDownload+23-1
#21Amit Kapila
amit.kapila16@gmail.com
In reply to: shihao zhong (#20)
#22Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: Amit Kapila (#21)
#23Manu
manuelreyesbravo@gmail.com
In reply to: Amit Kapila (#21)
#24shihao zhong
zhong950419@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#22)
#25shihao zhong
zhong950419@gmail.com
In reply to: Manu (#23)
#26Amit Kapila
amit.kapila16@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#22)
#27shveta malik
shveta.malik@gmail.com
In reply to: Amit Kapila (#21)
#28Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: shveta malik (#27)
#29Manu
manuelreyesbravo@gmail.com
In reply to: shihao zhong (#25)
#30Amit Kapila
amit.kapila16@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#28)
#31shveta malik
shveta.malik@gmail.com
In reply to: Amit Kapila (#30)
#32Hayato Kuroda (Fujitsu)
kuroda.hayato@fujitsu.com
In reply to: shveta malik (#31)
#33Zhijie Hou
houzhijie22@gmail.com
In reply to: shveta malik (#31)
#34Zhijie Hou
houzhijie22@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#32)
#35Amit Kapila
amit.kapila16@gmail.com
In reply to: Hayato Kuroda (Fujitsu) (#32)
#36Manu
manuelreyesbravo@gmail.com
In reply to: Amit Kapila (#21)
#37shihao zhong
zhong950419@gmail.com
In reply to: Manu (#36)
#38Manu
manuelreyesbravo@gmail.com
In reply to: shihao zhong (#37)
#39Alexander Lakhin
exclusion@gmail.com
In reply to: Manu (#38)
#40Manu
manuelreyesbravo@gmail.com
In reply to: Alexander Lakhin (#39)
#41Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Amit Kapila (#35)