Postmaster fails to shut down right after crash restart
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t51491psql -h localhost -U postgresBuilt from patchset v2 (message #2), October 23, 2026 at 06:32 AM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t51491_2 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t51491_2 && git checkout t51491_2Patchset v2 (message #2) is on t51491_2
Hello,
While developing a patch and running regression tests I noticed that the
postmaster could fail to shut down right after crash restart. It could
get stuck in the PM_WAIT_BACKENDS state forever.
As far as I understand, the problem occurs when a shutdown signal is
received before getting PMSIGNAL_RECOVERY_STARTED from the startup
process. In that case the FatalError flag is not cleared, and the
postmaster is stuck in PM_WAIT_BACKENDS waiting for the checkpointer,
which ignores SIGTERM.
To easily reproduce the problem I added pg_usleep in xlogrecovery.c just
before SendPostmasterSignal(PMSIGNAL_RECOVERY_STARTED). See the patch
attached.
Then I run a script that simulates a crash and does pg_ctl stop:
$ ./init.sh
[...]
$ ./stop-after-crash.sh
waiting for server to start.... done
server started
waiting for server to shut
down............................................................... failed
pg_ctl: server does not shut down
Some processes are still alive:
$ ps uf -C postgres
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
sergey 279874 0.0 0.0 222816 28560 ? Ss 14:25 0:00
/home/sergey/pgwork/devel/install/bin/postgres -D data
sergey 279887 0.0 0.0 222772 5664 ? Ss 14:25 0:00 \_
postgres: io worker 0
sergey 279888 0.0 0.0 222772 5664 ? Ss 14:25 0:00 \_
postgres: io worker 1
sergey 279889 0.0 0.0 222772 5664 ? Ss 14:25 0:00 \_
postgres: io worker 2
sergey 279891 0.0 0.0 222884 8480 ? Ss 14:25 0:00 \_
postgres: checkpointer
Here is an excerpt from the debug log:
postmaster[279874] LOG: all server processes terminated; reinitializing
startup[279890] LOG: database system was interrupted; last known up at
2025-04-24 14:25:58 MSK
startup[279890] LOG: database system was not properly shut down;
automatic recovery in progress
postmaster[279874] DEBUG: postmaster received shutdown request signal
postmaster[279874] LOG: received fast shutdown request
postmaster[279874] DEBUG: updating PMState from PM_STARTUP to
PM_STOP_BACKENDS
postmaster[279874] DEBUG: sending signal 15/SIGTERM to background
writer process with pid 279892
postmaster[279874] DEBUG: sending signal 15/SIGTERM to checkpointer
process with pid 279891
postmaster[279874] DEBUG: sending signal 15/SIGTERM to startup process
with pid 279890
postmaster[279874] DEBUG: sending signal 15/SIGTERM to io worker
process with pid 279889
postmaster[279874] DEBUG: sending signal 15/SIGTERM to io worker
process with pid 279888
postmaster[279874] DEBUG: sending signal 15/SIGTERM to io worker
process with pid 279887
postmaster[279874] DEBUG: updating PMState from PM_STOP_BACKENDS to
PM_WAIT_BACKENDS
startup[279890] LOG: invalid record length at 0/175A4D8: expected at
least 24, got 0
postmaster[279874] DEBUG: postmaster received pmsignal signal
startup[279890] LOG: redo is not required
checkpointer[279891] LOG: checkpoint starting: end-of-recovery
immediate wait
checkpointer[279891] LOG: checkpoint complete: wrote 0 buffers (0.0%),
wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled;
write=0.007 s, sync=0.002 s, total=0.026 s; sync files=2, longest=0.001
s, average=0.001 s; distance=0 kB, estimate=0 kB; lsn=0/175A4D8, redo
lsn=0/175A4D8
startup[279890] DEBUG: exit(0)
postmaster[279874] DEBUG: updating PMState from PM_WAIT_BACKENDS to
PM_WAIT_BACKENDS
checkpointer[279891] DEBUG: checkpoint skipped because system is idle
checkpointer[279891] DEBUG: checkpoint skipped because system is idle
I don't know how to fix this, but thought it's worth reporting.
Best regards,
--
Sergey Shinderuk https://postgrespro.com/
On 24 Apr 2025, at 17:06, Sergey Shinderuk <s.shinderuk@postgrespro.ru> wrote:
the postmaster could fail to shut down right after crash restart. It could get stuck in the PM_WAIT_BACKENDS state forever.
Hi Sergey,
I looked at this and reproduced it on current master. Your diagnosis looks
correct: if a smart or fast shutdown request arrives during crash-restart
reinitialization, before PMSIGNAL_RECOVERY_STARTED, FatalError remains
set. The normal shutdown path then sends SIGTERM to the processes started
for crash recovery and waits in PM_WAIT_BACKENDS, but the checkpointer
ignores SIGTERM.
PFA a three-patch series:
1. Allow wait injection points to be controlled through files under
pg_injection_points/ in the data directory.
2. Add a deterministic test for this bug, covering both smart and fast
shutdown.
3. Fix the shutdown path.
Michael, this looks like the concrete test case that was missing from
our July discussion [0]/messages/by-id/aher0VsjJ8xeNgLq@paquier.xyz. The test needs all three operations you
listed without relying on SQL: request attachment before triggering
crash restart so that the point is installed in the replacement shared
memory, observe that the startup process has reached it, and release
that process after the postmaster has handled the shutdown request.
Patch 0001 implements the filesystem approach you suggested. Creating
pg_injection_points/<point>/
causes the wait point to be attached when shared memory is initialized.
A process waiting there publishes
pg_injection_points/<point>/<pid>
and removing that file releases the process. The directories are
rescanned when shared memory is recreated after a crash, so the wait
point survives the exact reset that makes the SQL interface unusable
here. Wait points attached only through SQL do not create filesystem
state.
With patches 0001 and 0002, the test reproduces the hang. Patch 0003
handles a shutdown request received with FatalError set by using
HandleFatalError(PMQUIT_FOR_STOP, false). That sends the crash-recovery
children SIGQUIT instead of waiting for them after SIGTERM. There is no
graceful shutdown checkpoint to perform in the middle of crash recovery;
the next startup will run recovery again.
The filesystem interface for wait injection points is intended for HEAD.
The fix itself is independent and looks suitable for backpatching.
Thoughts?
Thank you!
Best regards, Andrey Borodin.
[0]: /messages/by-id/aher0VsjJ8xeNgLq@paquier.xyz
/messages/by-id/aher0VsjJ8xeNgLq@paquier.xyz
Attachments:
t51491_2v1-0001-injection_points-allow-driving-wait-points-throug.patchapplication/octet-stream; name=v1-0001-injection_points-allow-driving-wait-points-throug.patch; x-unix-mode=0644Download+441-7
v1-0002-Add-a-test-for-a-postmaster-shutdown-race-during-.patchapplication/octet-stream; name=v1-0002-Add-a-test-for-a-postmaster-shutdown-race-during-.patch; x-unix-mode=0644Download+167-1
v1-0003-Fix-postmaster-hang-on-shutdown-requested-during-.patchapplication/octet-stream; name=v1-0003-Fix-postmaster-hang-on-shutdown-requested-during-.patch; x-unix-mode=0644Download+31-5
I found an older thread about the same bug[0]/messages/by-id/ZWlrdQarrZvLsgIk@pryzbyj2023. Sorry I missed it before.
Justin Pryzby reported it in December 2023. Thomas Munro reproduced it
and identified the same window: FatalError is still set, the postmaster
has not yet processed PMSIGNAL_RECOVERY_STARTED, and shutdown gets stuck
waiting for the checkpointer. Thomas proposed a fix in that thread, and
Noah Misch later reproduced the problem and suggested clearing FatalError
when the replacement startup process is launched instead [1]/messages/by-id/20241025181701.ed.nmisch@google.com. There is
also an old CommitFest entry for Thomas's patch [2]https://commitfest.postgresql.org/patch/4884/.
The deterministic test in patch 0002 covers that report as well. Patch
0003 takes a third approach that only became directly available
recently. HandleFatalError() did not exist when Thomas posted his
patch; after it was introduced, it asserted that FatalError was not
already set. ead8f696b7cd removed that restriction. This lets 0003
retain FatalError during crash restart and handle a shutdown request in
that interval through HandleFatalError(PMQUIT_FOR_STOP, false). The
version from [2]https://commitfest.postgresql.org/patch/4884/ perhaps is easier to backpatch to 15-17 if we want to
backpatch.
Best regards, Andrey Borodin.
[0]: /messages/by-id/ZWlrdQarrZvLsgIk@pryzbyj2023
[1]: /messages/by-id/20241025181701.ed.nmisch@google.com
[2]: https://commitfest.postgresql.org/patch/4884/