Race conditions during parallel worker (unclean) exit
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253713psql -h localhost -U postgresBuilt from patchset v1 (message #1), September 09, 2026 at 03:51 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253713_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253713_1 && git checkout t253713_1Patchset v1 (message #1) is on t253713_1
In [1]/messages/by-id/32123.1788806780@localhost I noted that ERROR like "lost connection to ... worker" (currently
issued in ProcessParallelMessages()) might also be appropriate for REPACK
(CONCURRENTLY). While I'm still not sure about that, I suspect that something
is wrong about handling unclean exits in parallel.c.
According to the header comment, ParallelWorkerShutdown() should handle cases
like direct call of proc_exit(). By adding one like this
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index 17fcd246b0c..e8700ce8594 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -1385,6 +1385,7 @@ ParallelWorkerMain(Datum main_arg)
* Hooray! Primary initialization is complete. Now, we need to set up our
* backend-local state to match the original backend.
*/
+ proc_exit(1); /*
* Join locking group. We must do this before anything that could try to
and by running the following
begin;
create table a(i int);
set parallel_setup_cost to 0;
set parallel_tuple_cost to 0;
set min_parallel_table_scan_size to 0;
set min_parallel_index_scan_size to 0;
set max_parallel_workers_per_gather to 1;
set parallel_leader_participation to off;
set debug_parallel_query to 'regress';
table a;
rollback;
I can reproduce the expected behavior:
ERROR: lost connection to parallel worker
However, an additional sleep() call in ParallelWorkerShutdown()
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index 17fcd246b0c..1a05f16e0bc 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -1618,7 +1618,7 @@ ParallelWorkerShutdown(int code, Datum arg)
SendProcSignal(ParallelLeaderPid,
PROCSIG_PARALLEL_MESSAGE,
ParallelLeaderProcNumber);
-
+ sleep(1);
dsm_detach((dsm_segment *) DatumGetPointer(arg));
}appears to cause race conditions. What I see is that the leader process runs
ProcessParallelMessages() while the worker is still attached to the error
queue, so the leader gets SHM_MQ_WOULD_BLOCK when trying to read from the
queue. Thus the ERROR is not raised.
Moreover, the leader, after having seen the worker detached from the tuple
queue (TupleQueueReaderNext) a bit later, starts executor cleanup and, as the
worker does not send signals anymore, it gets stuck in
WaitForParallelWorkersToExit().
Attached is what might be a fix, but I'm not sure if the Terminate message is
appropriate even if the worker in fact didn't finish with success.
[1]: /messages/by-id/32123.1788806780@localhost
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
Hi Antonin,
Moreover, the leader, after having seen the worker detached from the tuple
queue (TupleQueueReaderNext) a bit later, starts executor cleanup and, as the
worker does not send signals anymore, it gets stuck in
WaitForParallelWorkersToExit().
There have been reports of leader hangs in parallel waits [1]/messages/by-id/20240920183931.f0.nmisch@google.com, but the one
I found was fixed by not entering parallel mode while interrupts were being
held.
That said, there may still be a good argument here to make the
abnormal-exit path
of a parallel worker more explicit, which your "sleep(1)" case demonstrates.
Attached is what might be a fix, but I'm not sure if the Terminate message is
appropriate even if the worker in fact didn't finish with success.
Right. I don't think "PqMsg_Terminate" is the right message here. It currently
represents normal completion:
```
/* Report success. */
pq_putmessage(PqMsg_Terminate, NULL, 0);
```
It seems cleaner to introduce a separate message for this case, maybe
"PqMsg_ParallelAborted", and keep the existing
"ERROR: lost connection to parallel worker" message.
[1]: /messages/by-id/20240920183931.f0.nmisch@google.com
--
Sami Imseih
Amazon Web Services (AWS)
Sami Imseih <samimseih.pg@gmail.com> wrote:
It seems cleaner to introduce a separate message for this case, maybe
"PqMsg_ParallelAborted", and keep the existing
"ERROR: lost connection to parallel worker" message.
I'm not sure this is worth adjusting the FE/BE protocol. I'd prefer regular
ERROR message (PqMsg_ErrorResponse), but only in some minimalistic form. The
worker would only send the message, w/o doing any error handling itself.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
It seems cleaner to introduce a separate message for this case, maybe
"PqMsg_ParallelAborted", and keep the existing
"ERROR: lost connection to parallel worker" message.I'm not sure this is worth adjusting the FE/BE protocol. I'd prefer regular
ERROR message (PqMsg_ErrorResponse), but only in some minimalistic form. The
worker would only send the message, w/o doing any error handling itself.
Perhaps PqMsg_ErrorResponse could work, although I am not clear what a
minimalistic
form means here. I was thinking of PqMsg_ParallelAborted because we
have precedent
already with PqMsg_Progress which was introduced in f1889729dd3a as
the 'P' message
for parallel index progress reported, but later converted to a macro
in a99cc6c6b4b.
(I think it was wrong to name it PqMsg_Progress, but that's besides
the point here).
--
Sami Imseih
Amazon Web Service (AWS)