[PATCH] Check interrupts while draining received WAL

Started by Nitin Jadhav6 days ago6 messageshackers
Jump to latest
#1Nitin Jadhav
nitinjadhavpostgres@gmail.com

Hi,

Following up on Andres's report [1]/messages/by-id/20210511022755.wcb5h4czadxsgewt@alap3.anarazel.de, the attached patch addresses
delayed SIGTERM processing while the WAL receiver drains continuously
available WAL.

WalReceiverMain() checks interrupts in its outer loop, but not in the
inner loop that processes all immediately available WAL. When the
receiver writes more slowly than the sender supplies WAL, it can
remain in that inner loop with SIGTERM pending. This can delay
receiver termination during server shutdown, promotion, or
reconfiguration. When termination is requested by startup, that
process can remain in ShutdownWalRcv(), visible as the WalReceiverExit
wait event.

The attached patch adds CHECK_FOR_INTERRUPTS() at the start of each
inner loop iteration so that shutdown requests are processed even when
WAL is continuously available.

For local reproduction, I queued approximately 128 MiB of WAL and used
temporary test-only instrumentation to sleep for 10 ms before each
receiver write, following Andres's suggestion [2]/messages/by-id/20210511025940.rkx73zezq3hq3eae@alap3.anarazel.de. While the receiver
was consuming that backlog, I stopped streaming by clearing
primary_conninfo and reloading on the standby:

ALTER SYSTEM SET primary_conninfo = '';
SELECT pg_reload_conf();

In the unfixed run, the receiver was still present after 5.068 seconds
and startup was waiting in WalReceiverExit. The receiver exited
naturally after 11.055 seconds. With the interrupt check and the same
instrumentation, receiver exit was observed after 0.033 seconds.

The artificial delay was used only for local reproduction and is not
included in the attached patch.

Comments and review would be appreciated

[1]: /messages/by-id/20210511022755.wcb5h4czadxsgewt@alap3.anarazel.de
[2]: /messages/by-id/20210511025940.rkx73zezq3hq3eae@alap3.anarazel.de

Best Regards,
Nitin Jadhav
Microsoft

Attachments:

t259235_1
v1-0001-Check-interrupts-while-draining-received-WAL.patchapplication/octet-stream; name=v1-0001-Check-interrupts-while-draining-received-WAL.patchDownload+2-0
#2Joao Rito
joaovictor.rito@gmail.com
In reply to: Nitin Jadhav (#1)
Re: [PATCH] Check interrupts while draining received WAL

Hi Nitin,

Thank you for the patch. I tested v1-0001 and it works for me. These
are my results.

Setup: master at 7fc2f0423320 (2026-11-16), Debian, gcc 14,
--enable-cassert --enable-tap-tests. The patch applies cleanly and
the build has no warnings.

For the reproduction I followed your description. I added a
test-only sleep of 10 ms before each XLogWalRcvWrite() call, read
from an environment variable. I queued about 124 MiB of WAL for the
standby and then acted while the receiver was still draining.
The same instrumentation is in both builds. Only the two lines of
your patch differ. Times are in milliseconds, 3 runs each:

action on the standby without patch with patch
clear primary_conninfo + reload 10812 10770 10929 33 29 40
pg_ctl stop -m fast 10435 10342 10337 104 108 105
pg_ctl promote 10554 10543 10664 117 115 114

Without the patch, the receiver exits only after it drains the
backlog, 10.3 to 10.9 s here. This delay comes from my test-only
sleep (124 MiB, 10 ms per write); it is not a measure of a real
system. With the patch it exits in 0.03 to 0.12 s.
My reload numbers match yours (0.033 s with the patch).

I also ran these recovery tests on the patched tree, without the
instrumentation: 001_stream_rep, 004_timeline_switch,
005_replay_delay and 013_crash_restart. All passed (100 tests).

On the idea: the outer loop already checks interrupts, so the same
check at the top of the inner loop looks right to me. By reading the
code, the check sits between messages, so no write is cut in half.
On exit, WalRcvDie() flushes the WAL already written, and a message
read but not yet written is dropped and sent again after restart. I
only read this; I did not test it.

A small remark. The patch has no test. I understand that a test
needs a way to slow the receiver, so I did not write one.

What I did not test:
- The full recovery suite, make check, isolation, meson, other
platforms.
- The backpatch. I used master only.
- A real slow receiver, without my sleep (real disk and sender).
- Exit while a message is already read but not written: I read the
code but did not test that the WAL is sent again.
- Any cost of the check on the hot path with a high WAL rate.
- Slots or recovery_min_apply_delay combined with this change.

Thanks again.

Best regards,
Joao Rito

#3Fujii Masao
masao.fujii@gmail.com
In reply to: Nitin Jadhav (#1)
Re: [PATCH] Check interrupts while draining received WAL

On Mon, Nov 16, 2026 at 5:59 PM Nitin Jadhav
<nitinjadhavpostgres@gmail.com> wrote:

The attached patch adds CHECK_FOR_INTERRUPTS() at the start of each
inner loop iteration so that shutdown requests are processed even when
WAL is continuously available.

Thanks for the patch! It looks good to me.

We should backpatch it to all supported versions? If so, for v17 and older,
ProcessWalRcvInterrupts() should be called instead of CHECK_FOR_INTERRUPTS()?

Regards,

--
Fujii Masao

#4Nitin Jadhav
nitinjadhavpostgres@gmail.com
In reply to: Fujii Masao (#3)
Re: [PATCH] Check interrupts while draining received WAL

Hi Fujii-San, Joao,

Thank you both for the review, and Joao for the detailed testing.

Thanks for the patch! It looks good to me.

We should backpatch it to all supported versions? If so, for v17 and older,
ProcessWalRcvInterrupts() should be called instead of CHECK_FOR_INTERRUPTS()?

Yes, I agree that this should be backpatched to all supported branches.

For v17 and older, ProcessWalRcvInterrupts() is the appropriate check.
Besides calling CHECK_FOR_INTERRUPTS(), it also handles the receiver’s
shutdown flag, so CHECK_FOR_INTERRUPTS() alone would not be
sufficient.

From v18 onward, the WAL receiver uses the standard die() SIGTERM
handler, which sets InterruptPending and ProcDiePending. Termination
is therefore handled through the normal interrupt machinery, making
CHECK_FOR_INTERRUPTS() sufficient.

Best Regards,
Nitin Jadhav
Microsoft

#5Fujii Masao
masao.fujii@gmail.com
In reply to: Nitin Jadhav (#4)
Re: [PATCH] Check interrupts while draining received WAL

On Wed, Nov 18, 2026 at 3:54 PM Nitin Jadhav
<nitinjadhavpostgres@gmail.com> wrote:

Yes, I agree that this should be backpatched to all supported branches.

OK, so I've pushed the patch and backpatched through v15. Thanks!

Regards,

--
Fujii Masao

#6Nitin Jadhav
nitinjadhavpostgres@gmail.com
In reply to: Fujii Masao (#5)
Re: [PATCH] Check interrupts while draining received WAL

OK, so I've pushed the patch and backpatched through v15. Thanks!

Thank you for committing and backpatching the fix so quickly!

Best Regards,
Nitin Jadhav
Microsoft