[PATCH v1] Fix races in Windows pthread emulation

Started by Harrison Booth4 months ago10 messageshackers
Jump to latest
#1Harrison Booth
harrisontbooth@gmail.com

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

The mutex initialization path used InterlockedExchange to set initstate to
2. A waiting thread could therefore change the fully initialized state from
1 back to 2. pthread_mutex_unlock would then see a state other than 1,
return EINVAL, and leave the critical section locked.

This patch uses InterlockedCompareExchange to claim initialization only
when initstate is 0. It also uses interlocked reads and publication for the
mutex state. The duplicated mutex implementations in ECPG and libpq receive
the same fix.

pthread_once previously used a Boolean state with plain reads and writes.
MSVC uses /volatile:iso on ARM64, so those accesses do not provide the
acquire and release ordering needed to publish initialized state. The patch
changes pthread_once_t to LONG and uses interlocked operations for every
read and publication.

I tested this against master at
1c9c35890421e96a91129b51f2c6446a6d95af95 on Windows 11 Pro ARM64 with
MSVC 19.44.35228. Both the native ARM64 build and the x64 build running
under ARM64 emulation compiled successfully. On each architecture:

- ecpg/ecpg passed 67 subtests.
- libpq/001_uri, 002_api, 003_load_balance_host_list, and 006_service
passed 241 subtests.

I did not add a new test because the existing ECPG thread/alloc test
exercises the failing path and reproduced the hang on native ARM64.

The mutex fast path now performs an interlocked read before entering or
leaving the critical section, and pthread_once checks do the same. I have
not benchmarked that cost. The patch changes no non-Windows code or
user-facing interface, so it requires no documentation change.

CommitFest entry 6314 has a pending patch that moves the libpq Windows
pthread implementation into src/port without changing this state machine.
This patch applies to current master. If that refactor lands first, the
libpq hunk will need a straightforward rebase.

This patch is intended for review and application to master.

Best,
Harrison

Attachments:

t253182_1
v1-0001-Fix-races-in-Windows-pthread-emulation.patchapplication/octet-stream; name=v1-0001-Fix-races-in-Windows-pthread-emulation.patchDownload+13-16
#2Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Harrison Booth (#1)
Re: [PATCH v1] Fix races in Windows pthread emulation

Hi,

Thank you for working on this!

On Fri, 24 Jul 2026 at 10:07, Harrison Booth <harrisontbooth@gmail.com> wrote:

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

I ran into this exact issue today [1]https://github.com/nbyavuz/postgres/actions/runs/32351446950/job/96371608653 and found your thread.

The mutex initialization path used InterlockedExchange to set initstate to
2. A waiting thread could therefore change the fully initialized state from
1 back to 2. pthread_mutex_unlock would then see a state other than 1,
return EINVAL, and leave the critical section locked.

I tried to understand the problem and the related Windows thread
functions, please correct my understanding:

1- In pthread_mutex_lock(), calling
InterlockedExchange(&mp->initstate, 2) writes unconditionally,
clobbering an initialized state 1 back to 2.
2- The thread enters the critical section while mp->initstate = 2.
3- Later, when pthread_mutex_unlock() is called, if (mp->initstate !=
1) evaluates to true, causing the function to return EINVAL without
releasing the underlying critical section, leading to a deadlock.

And your patch fixes this problem with
InterlockedCompareExchange(&mp->initstate, 2, 0), because now you
check the initial value so that you can't overwrite 1 with 2.

[1]: https://github.com/nbyavuz/postgres/actions/runs/32351446950/job/96371608653

--
Regards,
Nazir Bilal Yavuz
Microsoft

#3Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Nazir Bilal Yavuz (#2)
Re: [PATCH v1] Fix races in Windows pthread emulation

Hi,

On Thu, 20 Aug 2026 at 15:09, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Fri, 24 Jul 2026 at 10:07, Harrison Booth <harrisontbooth@gmail.com> wrote:

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

I ran into this exact issue today [1] and found your thread.

I hit that problem again today [1]https://github.com/nbyavuz/postgres/actions/runs/33748732194/job/100627756377#step:14:279. I created a Commitfest entry so we
don't lose track of this patch [2]https://commitfest.postgresql.org/patch/7255/.

[1]: https://github.com/nbyavuz/postgres/actions/runs/33748732194/job/100627756377#step:14:279
[2]: https://commitfest.postgresql.org/patch/7255/

--
Regards,
Nazir Bilal Yavuz
Microsoft

#4Heikki Linnakangas
hlinnaka@iki.fi
In reply to: Nazir Bilal Yavuz (#2)
Re: [PATCH v1] Fix races in Windows pthread emulation

On 20/08/2026 15:09, Nazir Bilal Yavuz wrote:

On Fri, 24 Jul 2026 at 10:07, Harrison Booth <harrisontbooth@gmail.com> wrote:

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

I ran into this exact issue today [1] and found your thread.

Is there some memory ordering reason why this only happens on ARM64?
AFAICS, it could happen on x86 too.

The mutex initialization path used InterlockedExchange to set initstate to
2. A waiting thread could therefore change the fully initialized state from
1 back to 2. pthread_mutex_unlock would then see a state other than 1,
return EINVAL, and leave the critical section locked.

I tried to understand the problem and the related Windows thread
functions, please correct my understanding:

1- In pthread_mutex_lock(), calling
InterlockedExchange(&mp->initstate, 2) writes unconditionally,
clobbering an initialized state 1 back to 2.
2- The thread enters the critical section while mp->initstate = 2.
3- Later, when pthread_mutex_unlock() is called, if (mp->initstate !=
1) evaluates to true, causing the function to return EINVAL without
releasing the underlying critical section, leading to a deadlock.

And your patch fixes this problem with
InterlockedCompareExchange(&mp->initstate, 2, 0), because now you
check the initial value so that you can't overwrite 1 with 2.

That works, but an InterlockedCompareExchange is more expensive than a
plain unlocked read. I presume the reason it didn't use
InterlockedCompareExchange in the first place. I think you could salvage
it by doing an unlocked check first (with a memory barrier?), and using
InterLockedCompareExchange only if the initialization is needed.

I don't think we need to care too much about performance of ecpg on
Windows -- I doubt anyone's doing anything that performance sensitive
with that combo -- but nevertheless.

-typedef bool pthread_once_t;
+typedef LONG pthread_once_t;

Is that safe to backpatch?

- Heikki

#5Alexander Lakhin
exclusion@gmail.com
In reply to: Heikki Linnakangas (#4)
Re: [PATCH v1] Fix races in Windows pthread emulation

Hello hackers,

05.10.2026 18:48, Heikki Linnakangas wrote:

On 20/08/2026 15:09, Nazir Bilal Yavuz wrote:

On Fri, 24 Jul 2026 at 10:07, Harrison Booth <harrisontbooth@gmail.com> wrote:

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

I ran into this exact issue today [1] and found your thread.

Is there some memory ordering reason why this only happens on ARM64? AFAICS, it could happen on x86 too.

FWIW, it also occurred on AMD64 animal hamerkop (twice in this year):
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamerkop&dt=2026-09-24%2014%3A35%3A01
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamerkop&dt=2026-03-26%2015%3A35%3A04 (ecpgCheck [fb072e1] (01:10:46))
(01:10:46))

I also managed to reproduce the hang locally, running 6 ecpg tests
concurrently on a Windows x86_64 VM, so I can test the patch if needed.

Best regards,
Alexander

#6Harrison Booth
harrisontbooth@gmail.com
In reply to: Alexander Lakhin (#5)
Re: [PATCH v1] Fix races in Windows pthread emulation

Thanks for the review and additional reproductions. ARM64 was where I
reproduced the hang; I didn’t mean to imply the mutex race was
ARM64-specific. I haven’t established the performance impact or
backpatch compatibility of the pthread_once change. Would it make
sense to separate that from the mutex fix? Alexander, thanks for
offering to test.

On Mon, 05 Oct 2026 21:00:00 +0300, Alexander Lakhin
<exclusion@gmail.com> wrote:

Show quoted text

Hello hackers,

05.10.2026 18:48, Heikki Linnakangas wrote:

On 20/08/2026 15:09, Nazir Bilal Yavuz wrote:

On Fri, 24 Jul 2026 at 10:07, Harrison Booth <harrisontbooth@gmail.com> wrote:

On native Windows ARM64, the ECPG thread/alloc test could hang until
Meson's 1000-second timeout. The cause was a race in the Windows pthread
mutex emulation.

I ran into this exact issue today [1] and found your thread.

Is there some memory ordering reason why this only happens on ARM64? AFAICS, it could happen on x86 too.

FWIW, it also occurred on AMD64 animal hamerkop (twice in this year):
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamerkop&dt=2026-09-24%2014%3A35%3A01
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamerkop&dt=2026-03-26%2015%3A35%3A04 (ecpgCheck [fb072e1]
(01:10:46))

I also managed to reproduce the hang locally, running 6 ecpg tests
concurrently on a Windows x86_64 VM, so I can test the patch if needed.

Best regards,
Alexander

#7Harrison Booth
harrisontbooth@gmail.com
In reply to: Alexander Lakhin (#5)
Re: [PATCH v2] Fix races in Windows pthread emulation

Attached is a mutex-only v2 rebased on master at 8e82d6610e7. It
leaves pthread_once and the existing type layouts unchanged.

Only a thread that claims state 0 initializes the critical section and
publishes state 1. The lock fast path uses an acquire load instead of
an interlocked read-modify-write; unlock uses a relaxed load after a
successful lock.

The deterministic reproducer returns EINVAL on the original code and
passes with v2. Stress testing passed 800,000 lock/unlock pairs per
configuration on native ARM64 MSVC, emulated x64 MSVC, and emulated
x64 MinGW.

Full ARM64 and x64 builds passed. On each architecture, ECPG passed 66
subtests, libpq passed 241, and pg_basebackup passed 151. The ARM64
ECPG rerun passed after adding the missing diff tool to PATH.

The patch applies cleanly to REL_17_STABLE and REL_18_STABLE; neither
stable branch was built.

Alexander, could you test this revision against your reproducer?
Heikki, do the acquire loads and separate pthread_once work address
your concerns?

A separate GitHub-hosted Windows x64 build and all nine test groups
passed with all three patches applied. ECPG passed 66 subtests, libpq
passed 241, and pg_basebackup passed 152.

Best,
Harrison

Attachments:

t253182_7
v2-0001-Fix-races-in-Windows-pthread-mutex-initialization.patchtext/x-diff; charset=utf-8; name=v2-0001-Fix-races-in-Windows-pthread-mutex-initialization.patchDownload+34-12
#8Heikki Linnakangas
hlinnaka@iki.fi
In reply to: Harrison Booth (#7)
Re: [PATCH v2] Fix races in Windows pthread emulation

On 13/10/2026 02:37, harrisontbooth@gmail.com wrote:

Heikki, do the acquire loads and separate pthread_once work address
your concerns?

Yeah.

+	/* Acquire the initialization state without a read-modify-write. */
+#ifdef _MSC_VER
+	if (ReadAcquire(&mp->initstate) != 1)
+#else
+	if (__atomic_load_n(&mp->initstate, __ATOMIC_ACQUIRE) != 1)
+#endif

Hmm, so ReadAcquire is only available on Visual Studio, and the
else-branch is for mingw builds? Is there no atomic read intrinsic
similar InterlockedCompareExchange() that would work on both?

- Heikki

#9Harrison Booth
harrisontbooth@gmail.com
In reply to: Heikki Linnakangas (#8)
Re: [PATCH v2] Fix races in Windows pthread emulation

Yes, the else branch covers GNU-style Windows builds. The current
mingw-w64 winnt.h doesn't provide ReadAcquire, so I used
__atomic_load_n there.

InterlockedCompareExchange(&mp->initstate, 0, 0) would provide a
common implementation with a full memory barrier. It would also put a
compare-exchange operation back on the initialized fast path. Avoiding
that was the reason for the split; I haven't benchmarked the
difference.

I haven't identified a common load-only Windows primitive in the
headers checked. I suggest keeping the load-only paths unless you'd
prefer the simpler common API path.

Best,

Harrison

Show quoted text

On Tue, 13 Oct 2026 11:03:21 +0300, Heikki Linnakangas <hlinnaka@iki.fi> wrote:

On 13/10/2026 02:37, harrisontbooth@gmail.com wrote:

Heikki, do the acquire loads and separate pthread_once work address
your concerns?

Yeah.

+	/* Acquire the initialization state without a read-modify-write. */
+#ifdef _MSC_VER
+	if (ReadAcquire(&mp->initstate) != 1)
+#else
+	if (__atomic_load_n(&mp->initstate, __ATOMIC_ACQUIRE) != 1)
+#endif

Hmm, so ReadAcquire is only available on Visual Studio, and the
else-branch is for mingw builds? Is there no atomic read intrinsic
similar InterlockedCompareExchange() that would work on both?

- Heikki

#10Alexander Lakhin
exclusion@gmail.com
In reply to: Harrison Booth (#7)
Re: [PATCH v2] Fix races in Windows pthread emulation

Hello Harrison,

13.10.2026 02:37, harrisontbooth@gmail.com wrote:

Alexander, could you test this revision against your reproducer?

I've tested your patch on master with 100 "test: thread/alloc" in
ecpg_schedule and 12 ecpg/ecpg tests running in parallel.

Without the patch, I got test failures due to timeouts on iterations
6, 59, 9, 8. With the patch applied, 300 iterations passed successfully
(twice).

Thank you for the fix!

Best regards,
Alexander