pg_dump: use threads for parallel workers on all platforms
pg_dump runs parallel workers in two completely different ways. On
non-Windows platforms, they're forked processes communicating with the
leader over pipes-- which is what pipes and processes are for. Windows
has no fork(), so there the workers are threads. But instead of
coordinating those threads as threads, the Windows port runs the same
process-based protocol on top of them, unchanged. Each worker's channel
to the leader is a loopback TCP socketpair on 127.0.0.1, opened when the
worker starts. So to tell a worker "dump table 1234," the leader
serializes the command to a string and writes it down that socket; the
worker reads it back a byte at a time, and the leader watches the
sockets with select() to see who's done. All of it to hand work to a
thread a few megabytes away in the same address space.
And because the leader waits on the workers with WaitForMultipleObjects,
which takes at most 64 handles, you can't run more than 64 jobs on
Windows. The parallelism limit there is a limit of the wait call. (The
non-Windows side doesn't have this limit-- it reaps workers with wait()
rather than WaitForMultipleObjects, so PG_MAX_JOBS is INT_MAX.)
None of this is broken; it works. It's threads pretending to be
processes because the code was written for processes, and the port kept
the protocol rather than rethinking it. I'd like to stop.
One model everywhere should be threads on all platforms, coordinated by
an in-process work queue-- a mutex and a couple of condition variables--
instead of two worker models bridged by an inter-process protocol.
To be clear, the unification is on the queue, not on what Windows does
today. Teaching the non-Windows side to talk to its own threads over a
socket, a byte at a time, would just be the same trick on more
platforms-- that's the part worth deleting, not copying.
I've done the Windows half, both to prove it out and because it's the
coordination layer the non-Windows side would adopt. The socket protocol
is gone; the leader hands work to a worker in memory instead of down a
loopback connection. The 64-job cap is gone. The unchecked
_beginthreadex return-- which on failure recorded a thread that didn't
exist as an idle worker-- is fixed. Dump and restore are byte-for-byte
identical to stock from -j2 through -j250. The non-Windows port to
threads isn't written yet, and I won't start it until the direction is
settled.
One piece I deliberately left for the unified version: the queue still
passes the command as a string-- "DUMP 1234"-- and the worker parses it
back and looks the ID up to recover the TocEntry it already came from.
In one address space that's ceremony; the queue could carry a {
T_Action, TocEntry * } and drop the serialize/parse/lookup entirely. I
didn't do it on Windows because the non-Windows side still forks, and a
pointer is meaningless across a process boundary. There, the string is
the serializer that path needs, so converting it to Windows-only would
add a second message format instead of removing one. With threading in
place across Windows and non-Windows, we can pass the work item directly
and delete buildWorkerCommand/parseWorkerCommand and the matching
response pair, plus the dumpId lookup, on every platform at once.
The cost is crash isolation. fork() gives each non-Windows worker its
own address space, so one that segfaults can't corrupt the leader or its
siblings; threads give that up. What it actually buys today is narrow.
The moment any worker dies, the leader pg_fatal()s and the whole dump
comes down, so processes don't give you recovery-- only the guarantee
that a corrupt worker can't scribble on a sibling's output before
everyone exits. Windows has run without even that for years. It's an
acceptable trade for a single implementation, but it's the real cost.
I'll say plainly that this fixes no user-visible bug, and nothing is
broken today. It's consolidation. There are two implementations of
parallel dump right now, and they drift: the Windows one grew the socket
emulation and the 64-job cap out of running a process protocol on
threads, and it carried bugs the non-Windows side never had, like the
_beginthreadex one above. One model means a fix or a feature lands once,
on a path exercised on every platform, instead of twice, with a seam
down the middle. Windows already shows the threaded model works here,
and threads are the half both sides can share, since Windows can't fork.
The Windows port originally kept the process shape to stay consistent
with the other platforms; this keeps that same goal, on the model that
actually ports.
The strongest evidence of the thread-safety of the worker-reachable
code-- Windows has run that path with threads for years. With the thread
rework, fmtId's static return value, is now _Thread_local. The global
state a worker reads is built during the single-threaded catalog phase,
before any worker exists.
Patches are attached-- 0001 and 0002 are independent and can be
committed separately; 0004 depends on 0003.
--
Bryan Green
EDB: https://www.enterprisedb.com
Attachments:
v1-0001-pg_dump-check-for-_beginthreadex-failure-in-paral.patchtext/plain; charset=UTF-8; name=v1-0001-pg_dump-check-for-_beginthreadex-failure-in-paral.patchDownload+2-1
v1-0002-Give-fmtId-s-temporary-buffer-thread-local-storag.patchtext/plain; charset=UTF-8; name=v1-0002-Give-fmtId-s-temporary-buffer-thread-local-storag.patchDownload+4-56
v1-0003-pg_dump-dispatch-parallel-workers-in-process-on-W.patchtext/plain; charset=UTF-8; name=v1-0003-pg_dump-dispatch-parallel-workers-in-process-on-W.patchDownload+167-31
v1-0004-pg_dump-allow-more-than-64-parallel-jobs-on-Windo.patchtext/plain; charset=UTF-8; name=v1-0004-pg_dump-allow-more-than-64-parallel-jobs-on-Windo.patchDownload+9-12
On 02/07/2026 19:30, Bryan Green wrote:
None of this is broken; it works. It's threads pretending to be
processes because the code was written for processes, and the port kept
the protocol rather than rethinking it. I'd like to stop.One model everywhere should be threads on all platforms, coordinated by
an in-process work queue-- a mutex and a couple of condition variables--
instead of two worker models bridged by an inter-process protocol.
+1
To be clear, the unification is on the queue, not on what Windows does
today. Teaching the non-Windows side to talk to its own threads over a
socket, a byte at a time, would just be the same trick on more
platforms-- that's the part worth deleting, not copying.I've done the Windows half, both to prove it out and because it's the
coordination layer the non-Windows side would adopt. ...
I see why you developed this that way, and it makes a lot of sense.
However, it has one downside: the Windows-only code cannot be tested
without Windows. At quick glance, it looks reasonable, but I'm a little
nervous committing more Windows-only code without being able to easily
play with it myself.
Once you have the final patch ready to switch non-Windows systems to
threaded model too, that gets easier.
With the thread rework, fmtId's static return value, is now
_Thread_local.
+1. This is the first _Thread_local in our codebase. It's in C11, so it
should just work, but we'll see if the buildfarm shows any surprises...
With this, the getLocalPQExpBuffer hook is never set. I think we can
just remove it, and rename defaultGetLocalPQExpBuffer() to
getLocalPQExpBuffer() directly.
I noticed that we currently call setFmtEncoding() in multiple places in
src/bin/pg_dump. I haven't looked at them closely, but I wonder if
there's some kind of thread-safety hazards there even without these patches.
And what about the 'quote_all_identifiers' global variable? Is that set
correctly in both models? I guess it's inherited through fork().
- Heikki
On 02/07/2026 19:30, Bryan Green wrote:
The unchecked _beginthreadex return-- which on failure recorded a
thread that didn't exist as an idle worker-- is fixed.
Committed and backpatched this patch (0001) now, to get that out of the
way. The rest remain. Thanks!
- Heikki
On 7/5/2026 6:21 PM, Heikki Linnakangas wrote:
On 02/07/2026 19:30, Bryan Green wrote:
None of this is broken; it works. It's threads pretending to be
processes because the code was written for processes, and the port kept
the protocol rather than rethinking it. I'd like to stop.One model everywhere should be threads on all platforms, coordinated by
an in-process work queue-- a mutex and a couple of condition variables--
instead of two worker models bridged by an inter-process protocol.+1
To be clear, the unification is on the queue, not on what Windows does
today. Teaching the non-Windows side to talk to its own threads over a
socket, a byte at a time, would just be the same trick on more
platforms-- that's the part worth deleting, not copying.I've done the Windows half, both to prove it out and because it's the
coordination layer the non-Windows side would adopt. ...I see why you developed this that way, and it makes a lot of sense.
However, it has one downside: the Windows-only code cannot be tested
without Windows. At quick glance, it looks reasonable, but I'm a little
nervous committing more Windows-only code without being able to easily
play with it myself.Once you have the final patch ready to switch non-Windows systems to
threaded model too, that gets easier.
I have reworked the patches: 0001-0002 are standalone for Windows.
0003-0005 are dependent on Thomas Munro's pthread.h. >> With the thread
rework, fmtId's static return value, is now
_Thread_local.
+1. This is the first _Thread_local in our codebase. It's in C11, so it
should just work, but we'll see if the buildfarm shows any surprises...With this, the getLocalPQExpBuffer hook is never set. I think we can
just remove it, and rename defaultGetLocalPQExpBuffer() to
getLocalPQExpBuffer() directly.
Agreed.
I noticed that we currently call setFmtEncoding() in multiple places in
src/bin/pg_dump. I haven't looked at them closely, but I wonder if
there's some kind of thread-safety hazards there even without these
patches.
There is fmtIdEncoding which is a plain static, and on Windows each dump
worker already calls setup_connection() -> setFmtEncoding() on its own
thread. Every writer stores the same value (the encoding is fixed for
the whole archive), so it's a same-value race. On the restore side the
leader sets it once (processEncodingEntry) and the workers only read it.
And what about the 'quote_all_identifiers' global variable? Is that set
correctly in both models? I guess it's inherited through fork().
getopt_long writes it during option parsing (--quote-all-identifiers
points straight at the global), before any worker or connection exists,
and nothing touches it again.
- Heikki
I expect changes to be needed. Thomas, I hope this is more of a help
than not.
0001-0002 stand alone and touch only Windows: fmtId()'s scratch buffer
becomes _Thread_local, and the Windows workers exchange work over an
in-process channel rather than a loopback socket driven by select().
They're useful on their own and don't depend on the rest.
0003-0005 do the unification on top of Thomas Munro's pg_threads.h, so
you will need his patch
[v1-0002-port-Provide-minimal-pg_threads.h-API.patch] to install
pg_threads API: 0003 ports the channel's locks to that API (mechanical,
no behavior change); 0004 switches all platforms to threads and deletes
the fork(), pgpipe() and select() code; 0005 drops the string command
protocol for plain structs, now that the leader and workers share an
address space.
Changes since v1:
- v1's _beginthreadex-failure check was committed separately as
75e201bf95, so it's dropped here.
- Dropped the standalone 64-worker-cap patch; 0004 removes that limit
for free by joining workers individually.
- Extended the series past Windows: 0003-0005 unify all platforms on
threads atop pg_threads.h.
- Per review, fmtId()'s getLocalPQExpBuffer indirection is removed in
0001 now that nothing overrides it.
--
Bryan Green
EDB: https://www.enterprisedb.com