BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount

Adding Robert and Andres because they know these areas best and can provide
some valuable insights to this issue.

On Fri, 17 Jul 2026 at 22:00, PG Bug reporting form <noreply@postgresql.org>
wrote:

The following bug has been logged on the website:

Bug reference: 19557
Logged by: Jacob Mørk Brejnbjerg
Email address: jacob@brejnbjerg.com
PostgreSQL version: 18.4
Operating system: Ubuntu 24.04 (kernel 6.8.0-124-generic)
Description:

A backend executing a parallel query has been stuck for 40+ hours in
wait_event IPC/ParallelFinish. All of its parallel workers exited cleanly
(exit code 0, nothing logged). pg_cancel_backend() and
pg_terminate_backend() have no effect. We kept the backend alive and
debugged it with gdb + debug symbols; the evidence shows a leaked
InterruptHoldoffCount, plus a design interaction that turns any such leak
into a permanently unkillable backend.

Possibly relevant configuration: io_method = io_uring,
max_parallel_workers_per_gather = 48,
shared_preload_libraries = 'timescaledb,pg_parquet'
(timescaledb 2.27.2, pg_parquet 0.5.1).
PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1).

The query is a COPY (SELECT <aggregation over one table> GROUP BY ...)
TO STDOUT (FORMAT parquet) issued via pg_parquet's COPY hook; the inner
select runs a parallel plan (Gather, 10+ workers planned) over
TimescaleDB compressed-hypertable chunks, i.e. an AIO-read-heavy scan.

Backtrace of the stuck leader (main thread; the only other thread is an
io_uring kernel worker, "iou-wrk-<pid>"):

#0 epoll_wait (epfd=754, events=..., maxevents=1, timeout=-1)
#1 WaitEventSetWait ()
#2 WaitLatch ()
#3 WaitForParallelWorkersToFinish ()
#4 ExecParallelFinish ()
#5-#8 (Gather shutdown path, static functions)
#9 standard_ExecutorRun ()
#10-#11 PortalRun ()
#12-#17 pg_parquet.so (COPY TO hook driving the portal)
#18 prev_ProcessUtility (timescaledb process_utility.c:124)
#19 timescaledb_ddl_command_start (...)
#20-#22 PortalRun ()
#23-#24 PostgresMain ()

Key state read from the live process via gdb:

InterruptPending = 1
ProcDiePending = 1 <- pg_terminate_backend arrived, never
serviced
QueryCancelPending = 1 <- pg_cancel_backend arrived, never serviced
InterruptHoldoffCount = 1 <- held, forever
CritSectionCount = 0
num_held_lwlocks = 0 <- NO LWLock is held
ParallelMessagePending = 1 <- worker messages queued, unprocessable

/proc signal state: SigPnd/ShdPnd are all zero - the SIGTERM/SIGINT were
delivered and consumed by the handlers (which set the flags above); the
backend wakes from WaitLatch, but CHECK_FOR_INTERRUPTS() is a no-op with
InterruptHoldoffCount > 0, so it just re-sleeps.

Why this is a permanent hang and not just an uncancellable wait:
worker-exit accounting is itself interrupt-driven. The workers' detach
notifications are only processed via CHECK_FOR_INTERRUPTS() ->
ProcessParallelMessages(), which is what clears each worker's error_mqh.
With the holdoff leaked, ProcessParallelMessages never runs
(ParallelMessagePending stays 1 - verified in the live process), so
WaitForParallelWorkersToFinish() believes the workers are still attached
and waits forever. The same leaked holdoff blocks ProcDiePending. One
leaked holdoff therefore produces both symptoms; only SIGKILL
(crash-restart) or a debugger can clear the backend.

On the origin of the leak: num_held_lwlocks = 0 rules out a stuck
LWLockAcquire, and the server log shows NO error from this backend for
its entire lifetime, which we believe rules out the
"ereport(ERROR) raised under HOLD, caught by PG_TRY" pattern (the raise
would have been logged by errfinish). That points to a straight-line
code path that called HOLD_INTERRUPTS() and returned without
RESUME_INTERRUPTS(). We audited the read path: bufmgr.c, read_stream.c,
shm_mq.c, condition_variable.c contain no bare holds; timescaledb 2.27.2
and pg_parquet 0.5.1 (and pgrx 0.16.1) contain no HOLD_INTERRUPTS at
all. The remaining bare holds on this workload's path are in
src/backend/storage/aio/aio.c (four HOLD/RESUME regions, including
pgaio_io_reclaim which runs completion callbacks under HOLD) and the
parallel-message path itself. Given io_method = io_uring and an
AIO-heavy scan we suspect an AIO path, but we could not identify the
exact site from the live process.

Frequency: once in ~6 days of heavy production use of this COPY path
(thousands of executions/day). Not deterministically reproducible so
far.

We have a gcore of the stuck backend and can keep it (and/or the live
process) available - happy to run additional gdb commands against it.

Separately, it may be worth considering a defensive measure: because
WaitForParallelWorkersToFinish depends on interrupt processing for its
own termination condition, any holdoff leak from any subsystem converts
into a silent, unkillable backend. Detecting "waiting on ParallelFinish
with INTERRUPTS_CAN_BE_PROCESSED() false and ParallelMessagePending set"
and at least logging (or treating it as an invariant violation) would
make this failure mode visible instead of silent.

--
Regards,
Rafia Sabih
CYBERTEC PostgreSQL International GmbH