libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

Started by Matthew Mochalkin6 days ago7 messageshackers
Jump to latest
#1Matthew Mochalkin
matthew.mochalkin@gmail.com

Hi,

A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 connections in parallel. The threads of one process get about the throughput of a single thread, and adding threads makes it worse. The same client as separate processes scales as expected. I ran into it with FrankenPHP (PHP's ZTS build served from the threads of one process, libpq 17.12, OpenSSL 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads once the role uses md5 or trust.

The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) does an implicit fetch of the digest, which takes process-wide locks inside libcrypto (openssl/openssl#19612 describes the same cost for EVP_DigestInit_ex). With one thread that is a per-iteration overhead; with several threads it is a lock convoy. The EVP_MAC conversion in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest parameter and has the same behaviour.

A 60-line C program with no libpq in it shows the effect. It runs the 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 CPUs, loops per second:

pattern per iteration 1 thr 8 thr 16 thr 8 proc
HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) 519 547 252 3,397 (libpq <= 17)
EVP_MAC_init(ctx, key, len, {digest=SHA256}) 600 1,191 547 4,582 (master)
HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150 5,684 9,832 6,366
HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) 1,997 9,272 12,095 18,557 (reuse key and digest)
EVP_MAC_init(ctx, NULL, 0, NULL) 1,085 8,045 13,785 13,613

And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 16, scram role:

role 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc
scram-sha-256 237 592 268 222 1,641 2,386
trust 595 1,882 3,814 559 2,640 3,594

This was discussed in February 2025 ("Optimize scram_SaltedPassword performance", /messages/by-id/Z6B7QXlPwPXewaQZ@paquier.xyz). That proposal was about single-thread latency and was dropped, partly because a faster PBKDF2 does not help the server much and partly because of the static state in the patch. The multithreaded case is different: it is not that the client is 4 times slower than it could be, it is that N threads are no faster than one, which for a threaded application server means the database connect rate of the whole process is capped around a few hundred per second. The fix does not need any static state: inside scram_SaltedPassword() the key is the same for every iteration, so the OpenSSL implementation can keep it in the context after the first pg_hmac_init() and re-init with a NULL key (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) on the EVP_MAC path). The fallback src/common/hmac.c can keep its current behaviour, or skip re-deriving the pads when the key pointer and length are unchanged.

I can send a patch for the OpenSSL path if there is interest. The reproduction (Docker, PHP and C) is at https://github.com/mattleads/frankenphp-pgconnect-repro.

Regards,
Matt Mochalkin

#2Daniel Gustafsson
daniel@yesql.se
In reply to: Matthew Mochalkin (#1)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

On 16 Nov 2026, at 22:00, Matthew Mochalkin <matthew.mochalkin@gmail.com> wrote:

I can send a patch for the OpenSSL path if there is interest.

There is always interest in modernizing our OpenSSL code, so please do.

--
Daniel Gustafsson

#3Matthew Mochalkin
matthew.mochalkin@gmail.com
In reply to: Daniel Gustafsson (#2)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

Thanks. Patch attached.

pg_hmac_init() accepts a NULL key, meaning "the key of the previous call on this context". The OpenSSL implementation re-initializes with EVP_MAC_init(ctx, NULL, 0, NULL) (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) before 3.0), which OpenSSL documents as reusing the existing key; its HMAC provider does exactly that internally. The fallback implementation already keeps the padded key in its context, so it only resets the running hash. Both return an error if no key was set before. scram_SaltedPassword() passes NULL from the second iteration on. No static state, no new function, and the existing code paths are untouched: 77 lines added, 2 changed.

Tests: src/test/authentication/t/001_password.pl passes with --with-ssl=openssl and without (216 tests each), so client and server proofs still agree in both implementations.

PQconnectdb()+PQfinish() in a loop against PostgreSQL 16 with a scram-sha-256 role, 8 CPUs, libpq built from master at 7fc2f04 with OpenSSL 3.5.7:

libpq 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc
Debian 17.12 (HMAC_Init_ex) 237 592 268 222 1,641 2,386
master 7fc2f04 (EVP_MAC_init) 271 1,096 707 281 1,910 2,479
master + patch 465 2,711 3,224 460 2,495 3,310

The single-thread gain is the one measured in the February thread (the proof no longer sets up the key and looks up the digest 4,096 times). The multi-thread row is the point of this patch: the threads of one process now scale like processes.

I can add it to the commitfest if you'd like it tracked there.

Regards,
Matt Mochalkin

Attachments:

t259262_3
v1-0001-Reuse-the-HMAC-key-across-the-iterations-of-scram_Sal.patchapplication/octet-stream; name=v1-0001-Reuse-the-HMAC-key-across-the-iterations-of-scram_Sal.patch; x-unix-mode=0644Download+77-3
#4Jakub Wartak
jakub.wartak@enterprisedb.com
In reply to: Matthew Mochalkin (#1)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

On Mon, Nov 16, 2026 at 11:08 PM Matthew Mochalkin
<matthew.mochalkin@gmail.com> wrote:

Hi,

A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 connections in parallel. The threads of one process get about the throughput of a single thread, and adding threads makes it worse. The same client as separate processes scales as expected. I ran into it with FrankenPHP (PHP's ZTS build served from the threads of one process, libpq 17.12, OpenSSL 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads once the role uses md5 or trust.

The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) does an implicit fetch of the digest, which takes process-wide locks inside libcrypto (openssl/openssl#19612 describes the same cost for EVP_DigestInit_ex). With one thread that is a per-iteration overhead; with several threads it is a lock convoy. The EVP_MAC conversion in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest parameter and has the same behaviour.

A 60-line C program with no libpq in it shows the effect. It runs the 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 CPUs, loops per second:

pattern per iteration 1 thr 8 thr 16 thr 8 proc
HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) 519 547 252 3,397 (libpq <= 17)
EVP_MAC_init(ctx, key, len, {digest=SHA256}) 600 1,191 547 4,582 (master)
HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150 5,684 9,832 6,366
HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) 1,997 9,272 12,095 18,557 (reuse key and digest)
EVP_MAC_init(ctx, NULL, 0, NULL) 1,085 8,045 13,785 13,613

And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 16, scram role:

role 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc
scram-sha-256 237 592 268 222 1,641 2,386
trust 595 1,882 3,814 559 2,640 3,594

This was discussed in February 2025 ("Optimize scram_SaltedPassword performance", /messages/by-id/Z6B7QXlPwPXewaQZ@paquier.xyz). That proposal was about single-thread latency and was dropped, partly because a faster PBKDF2 does not help the server much and partly because of the static state in the patch. The multithreaded case is different: it is not that the client is 4 times slower than it could be, it is that N threads are no faster than one, which for a threaded application server means the database connect rate of the whole process is capped around a few hundred per second. The fix does not need any static state: inside scram_SaltedPassword() the key is the same for every iteration, so the OpenSSL implementation can keep it in the context after the first pg_hmac_init() and re-init with a NULL key (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) on the EVP_MAC path). The fallback src/common/hmac.c can keep its current behaviour, or skip re-deriving the pads when the key pointer and length are unchanged.

I can send a patch for the OpenSSL path if there is interest. The reproduction (Docker, PHP and C) is at https://github.com/mattleads/frankenphp-pgconnect-repro.

To me it looks like another instance of [1]/messages/by-id/CAKZiRmw6jbShG+LJe0pY5G4w0ktoGwzPozsA=LhH_mN9wC+bhg@mail.gmail.com, isn't it? If that would be true
then pgbench --connect along with pgbouncer (quite to setup) could be enough
to test/verify your patch idea too.

[1]: /messages/by-id/CAKZiRmw6jbShG+LJe0pY5G4w0ktoGwzPozsA=LhH_mN9wC+bhg@mail.gmail.com

#5Matthew Mochalkin
matthew.mochalkin@gmail.com
In reply to: Jakub Wartak (#4)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

Yes, same thing. The contention your perf report showed inside HMAC_Init_ex() is what the patch removes.

I ran pgbench --connect through pgbouncer as you suggested. Same pgbench binary (master), base and patched libpq swapped with LD_LIBRARY_PATH. pgbouncer 1.24.1, auth_type = scram-sha-256, the role's SCRAM verifier in auth_file, PostgreSQL 16 behind it, OpenSSL 3.5.7, 8 CPUs for the client.

pgbench --connect -n -c N -j N -T 20 -f select1.sql, connections per second:

-c = -j base patched
1 403 451
8 1,170 5,380
16 734 8,555

With the base libpq pgbench used 6-8 cores for those ~1,000 connections/s, the "1000% CPU" you saw. With the patch it uses about the same CPU for 5-8x the connections.

If you’d like to repeat it. Put the SCRAM verifier in pgbouncer's auth_file, not the plain password: with the plain password pgbouncer does the PBKDF2 itself and caps the test at ~380 logins/s no matter what libpq does. And at 16 threads the patched client runs out of ephemeral ports in 20 s unless tcp_tw_reuse is on.

Regards,
Matt Mochalkin

Show quoted text

On 17 Nov 2026, at 5:24 PM, Jakub Wartak <jakub.wartak@enterprisedb.com> wrote:

On Mon, Nov 16, 2026 at 11:08 PM Matthew Mochalkin
<matthew.mochalkin@gmail.com <mailto:matthew.mochalkin@gmail.com>> wrote:

Hi,

A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 connections in parallel. The threads of one process get about the throughput of a single thread, and adding threads makes it worse. The same client as separate processes scales as expected. I ran into it with FrankenPHP (PHP's ZTS build served from the threads of one process, libpq 17.12, OpenSSL 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads once the role uses md5 or trust.

The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) does an implicit fetch of the digest, which takes process-wide locks inside libcrypto (openssl/openssl#19612 describes the same cost for EVP_DigestInit_ex). With one thread that is a per-iteration overhead; with several threads it is a lock convoy. The EVP_MAC conversion in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest parameter and has the same behaviour.

A 60-line C program with no libpq in it shows the effect. It runs the 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 CPUs, loops per second:

pattern per iteration 1 thr 8 thr 16 thr 8 proc
HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) 519 547 252 3,397 (libpq <= 17)
EVP_MAC_init(ctx, key, len, {digest=SHA256}) 600 1,191 547 4,582 (master)
HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150 5,684 9,832 6,366
HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) 1,997 9,272 12,095 18,557 (reuse key and digest)
EVP_MAC_init(ctx, NULL, 0, NULL) 1,085 8,045 13,785 13,613

And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 16, scram role:

role 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc
scram-sha-256 237 592 268 222 1,641 2,386
trust 595 1,882 3,814 559 2,640 3,594

This was discussed in February 2025 ("Optimize scram_SaltedPassword performance", /messages/by-id/Z6B7QXlPwPXewaQZ@paquier.xyz). That proposal was about single-thread latency and was dropped, partly because a faster PBKDF2 does not help the server much and partly because of the static state in the patch. The multithreaded case is different: it is not that the client is 4 times slower than it could be, it is that N threads are no faster than one, which for a threaded application server means the database connect rate of the whole process is capped around a few hundred per second. The fix does not need any static state: inside scram_SaltedPassword() the key is the same for every iteration, so the OpenSSL implementation can keep it in the context after the first pg_hmac_init() and re-init with a NULL key (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) on the EVP_MAC path). The fallback src/common/hmac.c can keep its current behaviour, or skip re-deriving the pads when the key pointer and length are unchanged.

I can send a patch for the OpenSSL path if there is interest. The reproduction (Docker, PHP and C) is at https://github.com/mattleads/frankenphp-pgconnect-repro.

To me it looks like another instance of [1], isn't it? If that would be true
then pgbench --connect along with pgbouncer (quite to setup) could be enough
to test/verify your patch idea too.

[1] - /messages/by-id/CAKZiRmw6jbShG+LJe0pY5G4w0ktoGwzPozsA=LhH_mN9wC+bhg@mail.gmail.com

#6Jakub Wartak
jakub.wartak@enterprisedb.com
In reply to: Matthew Mochalkin (#5)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

On Tue, Nov 17, 2026 at 5:37 PM Matthew Mochalkin
<matthew.mochalkin@gmail.com> wrote:

Yes, same thing. The contention your perf report showed inside HMAC_Init_ex() is what the patch removes.

I ran pgbench --connect through pgbouncer as you suggested. Same pgbench binary (master), base and patched libpq swapped with LD_LIBRARY_PATH. pgbouncer 1.24.1, auth_type = scram-sha-256, the role's SCRAM verifier in auth_file, PostgreSQL 16 behind it, OpenSSL 3.5.7, 8 CPUs for the client.

pgbench --connect -n -c N -j N -T 20 -f select1.sql, connections per second:

-c = -j base patched
1 403 451
8 1,170 5,380
16 734 8,555

With the base libpq pgbench used 6-8 cores for those ~1,000 connections/s, the "1000% CPU" you saw. With the patch it uses about the same CPU for 5-8x the connections.

Cool! Perhaps just please register the $thread using
https://commitfest.postgresql.org and mark it as 'Needs review', so it's not
lost and somebody can later pick it up (You probably need to register there to
do that).

-J.

#7Matthew Mochalkin
matthew.mochalkin@gmail.com
In reply to: Jakub Wartak (#6)
Re: libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration)

Cool! Perhaps just please register the $thread using
https://commitfest.postgresql.org <https://commitfest.postgresql.org/> and mark it as 'Needs review', so it's not
lost and somebody can later pick it up (You probably need to register there to
do that)..

Will do. My community account is a day old and commitfest.postgresql.orgi is still in its cool-off period for it, so I'll register the thread there as soon as it lets me in, with status Needs review.

Regards,
Matt