BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed
The following bug has been logged on the website:
Bug reference: 19411
Logged by: Matt Carter
Email address: mcarter@twosigma.com
PostgreSQL version: 16.7
Operating system: Debian GNU/Linux 12 with kernel 6.12.51-ts1-amd64
Description:
This report is for the libpq client library for PostgreSQL.
I noticed the issue because my Apache Airflow task supervisor processes were
experiencing memory growth of ~600 KB/min. When running hundreds of tasks,
my 32 GB machine OOM-crashed every few hours. I traced the problem to when
I upgraded libpq from version 13 to 16. Varying Python version, and
PostgreSQL DB version didn't make a difference. When I updated to psycopg3,
the leak rate dropped dramatically.
Confirmed leaking:
libpq 16.0.4
libpq 16.0.7
Confirmed NOT leaking:
libpq 13.0.11
libpq 13.11
My environment for the below reproducer is:
Client libpq: 16.0.4 or 16.0.7 (runtime version)
Python: 3.10.16 or 3.11.11
psycopg2: 2.9.10
SQLAlchemy: 1.4.46
PostgreSQL Server: 13.11 OR 16.7 (both affected equally)
Platform: Linux, K8s containers
Reliable reproducer:
```
#!/usr/bin/env python3
"""
Reproduces ~800 KB/min memory leak in libpq 16.x
MUST monitor with 'ps aux', NOT Python's resource module!
"""
import datetime, os, subprocess
from time import sleep
from sqlalchemy import Column, Integer, types, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.pool import NullPool
# Verify libpq version
import psycopg2
print(f"psycopg2 compiled against libpq: {psycopg2.__libpq_version__}")
# Check ACTUAL runtime libpq (critical!)
so_path = os.path.dirname(psycopg2.__file__) +
"/_psycopg.cpython-311-x86_64-linux-gnu.so"
result = subprocess.run(["ldd", so_path], capture_output=True, text=True)
for line in result.stdout.split('\n'):
if 'libpq.so' in line:
print(f"ACTUAL runtime libpq:
{line.split('=>')[1].split('(')[0].strip()}")
print(f"\nMonitor this PID: {os.getpid()}")
print("Run in another terminal: watch -n 10 'ps -p <PID> -o
pid,rss,vsz,cmd'")
print()
# Setup SQLAlchemy with NullPool (creates/destroys connections)
Base = declarative_base(metadata=None)
class TestTable(Base):
__tablename__ = "test_leak_table"
id = Column(Integer, primary_key=True)
updated_at = Column(types.DateTime(timezone=True))
obj = TestTable()
engine = create_engine(
"postgresql://your-server/your-database",
poolclass=NullPool, # Critical: no pooling, create/destroy each time
encoding="utf-8"
)
Session = scoped_session(sessionmaker(
autocommit=False,
autoflush=False,
bind=engine,
expire_on_commit=False,
))
session = Session()
session.add(obj)
# Monitor memory using ps aux (CRITICAL - internal monitoring doesn't show
leak!)
def get_rss_kb():
"""Get RSS from ps aux - DO NOT use resource.getrusage()!"""
pid = os.getpid()
result = subprocess.run(["ps", "aux"], capture_output=True, text=True)
for line in result.stdout.split('\n'):
if str(pid) in line and 'python' in line:
return int(line.split()[5]) # RSS is 6th column
return None
print("Starting 10-minute test...")
print(f"{'Time(s)':<8} | {'Iteration':<10} | {'RSS(KB)':<10} |
{'Growth(KB)':<10}")
print("-" * 55)
start_time = datetime.datetime.now()
baseline_rss = None
for i in range(1, 601): # 600 iterations = 10 minutes
# Update and commit (creates/destroys connection with NullPool)
obj.updated_at = datetime.datetime.now(datetime.timezone.utc)
session.commit()
# Report every 30 iterations
if i % 30 == 0:
rss = get_rss_kb()
if rss:
if i == 30:
baseline_rss = rss
growth = 0
else:
growth = rss - baseline_rss
elapsed = (datetime.datetime.now() - start_time).total_seconds()
print(f"{int(elapsed):<8} | {i:<10} | {rss:<10} | {growth:<10}",
flush=True)
sleep(1)
# Final results
final_rss = get_rss_kb()
total_elapsed = (datetime.datetime.now() - start_time).total_seconds()
growth_total = final_rss - baseline_rss
leak_rate = growth_total / (total_elapsed / 60)
print()
print("=" * 60)
print(f"Baseline RSS (t=30s): {baseline_rss} KB")
print(f"Final RSS (t=600s): {final_rss} KB")
print(f"Total Growth: {growth_total} KB")
print(f"Leak Rate: {leak_rate:.1f} KB/min")
print("=" * 60)
```
The DB set-up for this reproducer is simple:
```
CREATE TABLE test_leak_table (
id SERIAL PRIMARY KEY,
updated_at TIMESTAMP WITH TIME ZONE
);
INSERT INTO test_leak_table (updated_at) VALUES (NOW());
```
When I run this with libpq 13.x, I get:
Baseline: ~41,000 KB
Final: ~41,200 KB
Growth: ~200 KB (stabilizes)
Leak Rate: ~14 KB/min
But when I run it with libpq 16.x, I reproduce the leak behavior:
Baseline: ~42,000 KB
Final: ~52,000 KB
Growth: ~10,000 KB (continuous linear growth)
Leak Rate: ~800 KB/min
Notes:
1. PostgreSQL server version does NOT matter
PG 13.11: ~797 KB/min average
PG 16.7: ~802 KB/min average
Difference: <1% (within measurement error)
2. Leak is linear and continuous
Grows ~530 KB every 30 seconds
No stabilization
No sawtooth pattern
Suggests unbounded memory accumulation
3. Leak is in C Library Memory
The leak is:
- Visible in `ps aux` RSS (operating system view)
- NOT visible in Python's resource.getrusage() (application view)
- NOT any Python object (I checked with the Python VM's `gc.get_objects()`)
This indicates the leak is in libpq's C-level allocations, not Python heap.
4. The leak requires connection churn. In particular, it needs:
- NullPool (no connection pooling)
- Repeated connection create/destroy
- UPDATE + COMMIT operations each iteration
Note that the only psycopg2 version I tested with was 2.9.10, as it is the
only version available in my test environment.
Also, I only tested with SQLAlchemy version 1.4.46, as v2 would require a
rewrite due to an incompatible API.
Workarounds:
Use libpq 13.x (43x improvement: 14-22 KB/min vs 800 KB/min).
Use connection pooling to reduce connection churn.
Use psycopg3.
Recycle processes frequently.
PG Bug reporting form <noreply@postgresql.org> writes:
This report is for the libpq client library for PostgreSQL.
I noticed the issue because my Apache Airflow task supervisor processes were
experiencing memory growth of ~600 KB/min. When running hundreds of tasks,
my 32 GB machine OOM-crashed every few hours. I traced the problem to when
I upgraded libpq from version 13 to 16. Varying Python version, and
PostgreSQL DB version didn't make a difference. When I updated to psycopg3,
the leak rate dropped dramatically.
I think there are way too many moving parts here, and too few
configuration details, to allow assigning blame confidently.
I tried making a simple C test program that just connected, issued
BEGIN/UPDATE/COMMIT, and disconnected in a tight loop. I see zero
leakage with that. So either the leak isn't actually in libpq, or
there's some critical environmental factor you didn't mention.
Plausible such factors include connection type and authentication
method. (For example, years ago libpq did leak memory while using
GSSAPI encryption.) I tried both regular and SSL connections but
didn't really push hard on that, since I'd just be guessing blindly
about what your setup is.
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> wrote:
I think there are way too many moving parts here, and too few configuration details,
to allow assigning blame confidently.
I tried making a simple C test program that just connected, issued
BEGIN/UPDATE/COMMIT, and disconnected in a tight loop. I see zero leakage with
that. So either the leak isn't actually in libpq, or there's some critical environmental
factor you didn't mention.Plausible such factors include connection type and authentication method. (For
example, years ago libpq did leak memory while using GSSAPI encryption.) I tried
both regular and SSL connections but didn't really push hard on that, since I'd just
be guessing blindly about what your setup is.
Tom,
Thank you for taking the time to test this and for the feedback. Your C test showing no leak suggests the issue is specific to how psycopg2 uses libpq, not libpq itself. I apologize for not including enough environmental details. I used Kerberos/GSSAPI with SSL (TLS 1.2 connections). My connection string was: "postgresql://hostname/database" (no password, Kerberos auth).
Your mention of "years ago libpq did leak memory while using GSSAPI encryption" is interesting because we ARE using GSSAPI/Kerberos authentication.
Given your C test showed no leak, this appears to be specific to GSSAPI authentication, or perhaps specific to how psycopg2 uses libpq with GSSAPI.
I can test with non-GSSAPI authentication to try to isolate that variable. I can also create a pure psycopg2 reproducer (without SQLAlchemy). I can also test whether disabling GSSAPI encryption (but keeping GSSAPI auth) changes the behavior. Would testing with GSSAPI authentication help narrow this down? I can also report this to the psycopg2 project if you think it's their issue.
Thanks again for your help,
Matt
Matt Carter <Matt.Carter@twosigma.com> writes:
Tom Lane <tgl@sss.pgh.pa.us> wrote:
I think there are way too many moving parts here, and too few configuration details,
to allow assigning blame confidently.
Thank you for taking the time to test this and for the feedback. Your C test showing no leak suggests the issue is specific to how psycopg2 uses libpq, not libpq itself. I apologize for not including enough environmental details. I used Kerberos/GSSAPI with SSL (TLS 1.2 connections). My connection string was: "postgresql://hostname/database" (no password, Kerberos auth).
Your mention of "years ago libpq did leak memory while using GSSAPI encryption" is interesting because we ARE using GSSAPI/Kerberos authentication.
Interesting. I wondered about GSSAPI, but spinning up such an
environment is more work than I wanted to do on speculation.
I can test with non-GSSAPI authentication to try to isolate that variable. I can also create a pure psycopg2 reproducer (without SQLAlchemy). I can also test whether disabling GSSAPI encryption (but keeping GSSAPI auth) changes the behavior. Would testing with GSSAPI authentication help narrow this down? I can also report this to the psycopg2 project if you think it's their issue.
Please try varying the connection type and encryption. I do suspect
this may be psycopg2's fault, but we lack enough data to pin blame
as yet.
regards, tom lane
Hi,
On 2026-02-17 13:47:49 -0500, Tom Lane wrote:
Matt Carter <Matt.Carter@twosigma.com> writes:
Thank you for taking the time to test this and for the feedback. Your C test showing no leak suggests the issue is specific to how psycopg2 uses libpq, not libpq itself. I apologize for not including enough environmental details. I used Kerberos/GSSAPI with SSL (TLS 1.2 connections). My connection string was: "postgresql://hostname/database" (no password, Kerberos auth).
Your mention of "years ago libpq did leak memory while using GSSAPI encryption" is interesting because we ARE using GSSAPI/Kerberos authentication.Interesting. I wondered about GSSAPI, but spinning up such an
environment is more work than I wanted to do on speculation.
Heh, understandable...
I can test with non-GSSAPI authentication to try to isolate that variable. I can also create a pure psycopg2 reproducer (without SQLAlchemy). I can also test whether disabling GSSAPI encryption (but keeping GSSAPI auth) changes the behavior. Would testing with GSSAPI authentication help narrow this down? I can also report this to the psycopg2 project if you think it's their issue.
Please try varying the connection type and encryption. I do suspect
this may be psycopg2's fault, but we lack enough data to pin blame
as yet.
Matt, you could try analyzing the memory usage with heaptrack, it tends to be
pretty good at finding them even in uninstrumented builds, as long as enough
debug symbols for a backtrace are available. Often enough it'll pinpoint
where the leak is coming from quite easily (but note that it will report some
constant-sized leaks that are "intentional").
Greetings,
Andres
Tom Lane <tgl@sss.pgh.pa.us> wrote:
Please try varying the connection type and encryption.
Good idea. Here's what I found:
Test 1:
Configuration: Original (SQLAlchemy + NullPool)
libpq: 16.0.7
SQLAlchemy: Yes
Leak Rate: 803 KB/min
Test 2:
Configuration: Pure psycopg2 (default)
libpq: 16.0.7
SQLAlchemy: NO
Leak Rate: 801 KB/min
Test 3:
Configuration: Pure psycopg2 + gssencmode=disable
libpq: 16.0.7
SQLAlchemy: NO
Leak Rate: 858 KB/min
Test 4:
Configuration: Pure psycopg2 + SSL only
libpq: 16.0.7
SQLAlchemy: NO
Leak Rate: 861 KB/min
Test 5:
Configuration: Pure psycopg2
libpq: 13.0.11
SQLAlchemy: NO
Leak Rate: 17 KB/min
So, it seems none of these changes avoids the leak:
- Removing SQLAlchemy
- Disabling GSSAPI encryption
- Using SSL-only
- Changing PostgreSQL DB Server version from 16 to 13
The only changes that I found that avoid the leak are:
- Changing libpq version from 16 to 13, or
- Changing psycopg version from 2 to 3.
Tom, since your C test showed no leak, the issue is likely in how psycopg2 calls libpq, not pure libpq itself.
I guess I should report this to the psycopg2 project.
Matt Carter wrote:
Tom, since your C test showed no leak, the issue is likely in how psycopg2 calls libpq, not pure libpq itself.
FYI, I have reported the issue to the psycopg project:
https://github.com/psycopg/psycopg2/issues/1827
Thanks you for your help, Tom. Your test results helped show that this is likely a psycopg2 issue.
Cheers,
Matt
Tom Lane wrote:
Interesting. I wondered about GSSAPI, but spinning up such an
environment is more work than I wanted to do on speculation.
Your suspicion was right, and I'm sorry for pointing at psycopg2 earlier. The leak is in libpq, and it needs GSSAPI *authentication*. Every connection to my server authenticates via "gss" in pg_hba.conf. My gssencmode=disable runs only turned off GSS encryption, so none of my tests ever removed it.
pg_GSS_continue() runs once per GSS round trip, and Kerberos with mutual authentication takes two. On every call it does
/* Only try to acquire credentials if GSS delegation isn't disabled. */
if (!pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;
pg_GSS_have_cred_cache() calls gss_acquire_cred() and stores the result in *cred_out. It doesn't release the credential that is already there. pqDropConnection() releases only the last one, so the credential from the first round trip is orphaned. With MIT krb5 that costs about 17.8 kB per connection, mostly 4144-byte keytab buffers allocated under gss_acquire_cred().
This is gdb on libpq 16.7 with gssencmode=disable, breaking on the GSSAPI entry points. Every connection looks like this:
PQconnectdb
gss_acquire_cred from pg_GSS_have_cred_cache
gss_init_sec_context cred=...b580 from pg_GSS_continue
gss_acquire_cred from pg_GSS_have_cred_cache
gss_init_sec_context cred=...85c0 from pg_GSS_continue
PQfinish
gss_release_cred cred=...85c0 from pqDropConnection
...b580 is never released.
The other callers already guard against this. For example, fe-secure-gssapi.c calls pg_GSS_have_cred_cache() only when conn->gcred == GSS_C_NO_CREDENTIAL. The patch below does the same in pg_GSS_continue(). It's against 18.3 and applies to 16.7 with only a line offset:
--- a/src/interfaces/libpq/fe-auth.c
+++ b/src/interfaces/libpq/fe-auth.c
@@ -101,8 +101,13 @@
if (conn->Pfdebug)
pqTraceOutputMessage(conn, conn->inBuffer + conn->inStart, false);
- /* Only try to acquire credentials if GSS delegation isn't disabled. */
- if (!pg_GSS_have_cred_cache(&conn->gcred))
+ /*
+ * Acquire credentials on the first call only. Each GSS round trip comes
+ * through here, and acquiring again would leak the credential already in
+ * conn->gcred.
+ */
+ if (conn->gcred == GSS_C_NO_CREDENTIAL &&
+ !pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;
if (conn->gssdelegation && conn->gssdelegation[0] == '1')
I built libpq from the 16.7 and 18.3 release tarballs, with MIT krb5 1.20.1 and OpenSSL 1.1, both with and without the patch. I then ran the reproducer below against a server using gss auth + SSL. These are the bytes per connection still allocated after warm-up, over 450 connections:
unpatched patched
16.7 17823 123
18.3 17844 98
With gssencmode=disable, the patched builds measure 31 and 74 bytes/connection. Connections still authenticate normally. With the patch, gdb shows one gss_acquire_cred() per connection, used for both gss_init_sec_context() calls and released in pqDropConnection(). Over 3,000 connections (least-squares slope after warm-up), the patched builds still grow by 25-31 bytes per connection. libpq 13.11 on the same server grows by 18 bytes per connection, so that remainder doesn't look new in v16.
The unpatched leak also shows up in libpq 17.6 as bundled in psycopg-binary 3.2.13, with MIT krb5 1.15.1: 17985 bytes/connection. libpq 13.11 and 15 don't have it, which fits this code arriving with GSS credential delegation in v16. I haven't tested master.
Andres: thanks for the heaptrack pointer. An allocation tracker with backtraces put 95% of the retained memory under gss_acquire_cred() called from libpq, which is what led here.
--- gss_leak_repro.c ---
/*
* Build: cc -o gss_leak_repro gss_leak_repro.c -I$(pg_config --includedir) \
* -L$(pg_config --libdir) -lpq
* Run: kinit; ./gss_leak_repro "host=<server with gss in pg_hba.conf> dbname=postgres" 1000
* Needs glibc >= 2.33 for mallinfo2().
*/
#include <malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>
static size_t
heap_in_use(void)
{
struct mallinfo2 mi = mallinfo2();
return mi.uordblks + mi.hblkhd;
}
int
main(int argc, char **argv)
{
const char *conninfo = argc > 1 ? argv[1] : "dbname=postgres";
int n = argc > 2 ? atoi(argv[2]) : 1000;
int warmup = n / 10;
size_t base = 0;
printf("libpq %d\n", PQlibVersion());
for (int i = 1; i <= n; i++)
{
PGconn *conn = PQconnectdb(conninfo);
if (PQstatus(conn) != CONNECTION_OK)
{
fprintf(stderr, "%s", PQerrorMessage(conn));
return 1;
}
if (i == 1)
printf("GSSAPI auth used: %d\n", PQconnectionUsedGSSAPI(conn));
PQfinish(conn);
if (i == warmup)
base = heap_in_use();
}
printf("%.1f bytes per connection still allocated (%d connections after warm-up)\n",
(double) (heap_in_use() - base) / (n - warmup), n - warmup);
return 0;
}
--- end ---
Matt
Matt Carter <Matt.Carter@twosigma.com> writes:
Your suspicion was right, and I'm sorry for pointing at psycopg2 earlier. The leak is in libpq, and it needs GSSAPI *authentication*. Every connection to my server authenticates via "gss" in pg_hba.conf. My gssencmode=disable runs only turned off GSS encryption, so none of my tests ever removed it.
pg_GSS_continue() runs once per GSS round trip, and Kerberos with mutual authentication takes two. On every call it does
/* Only try to acquire credentials if GSS delegation isn't disabled. */
if (!pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;
pg_GSS_have_cred_cache() calls gss_acquire_cred() and stores the result in *cred_out. It doesn't release the credential that is already there.
Ooops. It looks like this isn't the only place that makes that
mistake: select_next_encryption_method seems at risk too, or at least
if it's not, that's because of caller logic not anything it's doing
itself. I think that's partially because pg_GSS_have_cred_cache is
poorly named: it sounds like a status test not an action, and its API
is just bizarre. I propose changing its name altogether, as attached.
regards, tom lane
Import Notes
Reply to msg id not found: BL3PR08MB72830323C3ED904202F002E0996E2@BL3PR08MB7283.namprd08.prod.outlook.com
Tom Lane wrote:
I propose changing its name altogether, as attached.
Thanks, Tom. I tested your patch against my GSSAPI-authenticated server (MIT krb5 1.20.1, OpenSSL 1.1). It applies to 18.3 with line offsets and builds without warnings. Using the reproducer from my previous message, these are the bytes per connection still allocated after warm-up (450 connections):
18.3 unpatched 17844
18.3 + your patch 108
18.3 + your patch, gssencmode=disable 42
GSSAPI authentication works as before. Under gdb, every gss_acquire_cred() now has a matching gss_release_cred():
- With gssencmode=disable there is one acquire per connection. Both gss_init_sec_context() calls share it, and pqDropConnection() releases it.
- With the default gssencmode, pqsecure_open_gss() releases the encryption probe's credential before the fallback. Authentication then acquires one more, and pqDropConnection() releases that.
I couldn't exercise gssencmode=require past negotiation, because my server's pg_hba.conf rejects GSSAPI-encrypted connections. It fails the same way with and without the patch.
We would benefit from back-patches to 16 and 17:
- Our Airflow deployments get libpq 17.11 from psycopg2-binary 2.9.13, which measures 17930 bytes/connection with the same reproducer.
- Our internal builds ship 16.x. There the fe-connect.c hunks won't apply as-is, but the pg_GSS_continue() change alone takes 16.7 from 17823 to 123 bytes per connection.
Matt
________________________________________
From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Thursday, 29 October 2026 20:47:18
To: Matt Carter
Cc: Andres Freund; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed
Matt Carter <Matt.Carter@twosigma.com> writes:
Your suspicion was right, and I'm sorry for pointing at psycopg2 earlier. The leak is in libpq, and it needs GSSAPI *authentication*. Every connection to my server authenticates via "gss" in pg_hba.conf. My gssencmode=disable runs only turned off GSS encryption, so none of my tests ever removed it.
pg_GSS_continue() runs once per GSS round trip, and Kerberos with mutual authentication takes two. On every call it does
/* Only try to acquire credentials if GSS delegation isn't disabled. */
if (!pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;
pg_GSS_have_cred_cache() calls gss_acquire_cred() and stores the result in *cred_out. It doesn't release the credential that is already there.
Ooops. It looks like this isn't the only place that makes that
mistake: select_next_encryption_method seems at risk too, or at least
if it's not, that's because of caller logic not anything it's doing
itself. I think that's partially because pg_GSS_have_cred_cache is
poorly named: it sounds like a status test not an action, and its API
is just bizarre. I propose changing its name altogether, as attached.
regards, tom lane
Matt Carter <Matt.Carter@twosigma.com> writes:
Tom Lane wrote:
I propose changing its name altogether, as attached.
Thanks, Tom. I tested your patch against my GSSAPI-authenticated server (MIT krb5 1.20.1, OpenSSL 1.1). It applies to 18.3 with line offsets and builds without warnings. Using the reproducer from my previous message, these are the bytes per connection still allocated after warm-up (450 connections):
18.3 unpatched 17844
18.3 + your patch 108
18.3 + your patch, gssencmode=disable 42
GSSAPI authentication works as before.
Thanks for testing that, but after looking closer, I believe that
pg_GSS_continue has even more problems here. It will attempt to
acquire credentials whether or not GSS delegation is enabled,
contrary to the adjacent comment and unlike what happens in
pqsecure_open_gss. And if GSS delegation is enabled, it will
set the GSS_C_DELEG_FLAG whether or not it was able to acquire
credentials, which seems not very sane and again different from
the logic in pqsecure_open_gss. So I think what we really need
is more like the attached. This passes our own test suite, but
I wonder if you can exercise it in your own environment.
We would benefit from back-patches to 16 and 17:
Yeah, this problem was visibly introduced in v16 when we added
GSS delegation support, so I anticipate back-patching that far.
regards, tom lane
Tom Lane wrote:
So I think what we really need is more like the attached. This passes
our own test suite, but I wonder if you can exercise it in your own
environment.
v1 works here. It applies to 18.3 with line offsets and builds without warnings. These are bytes per connection still allocated after warm-up (450 connections), against the same server (gss auth + SSL, MIT krb5 1.20.1):
18.3 unpatched 17844
v1, defaults 94
v1, gssencmode=disable 15
v1, gssdelegation=1 45
v1, gssdelegation=1 gssencmode=disable 46
A 3,000-connection run with defaults has a least-squares slope of 13.5 bytes/connection after warm-up. libpq 13.11 measures 17.5 on the same server, so v1 is at or below pre-16 behavior. Every connection authenticated with GSSAPI.
gdb on gss_init_sec_context() matches your description:
- With gssdelegation unset, the authentication path never calls gss_acquire_cred(). Both gss_init_sec_context() calls get GSS_C_NO_CREDENTIAL with req_flags = GSS_C_MUTUAL_FLAG.
- With gssdelegation=1, there is one gss_acquire_cred() per connection. Both calls share it with req_flags = GSS_C_MUTUAL_FLAG | GSS_C_DELEG_FLAG, and pqDropConnection() releases it.
- With the default gssencmode, the encryption probe's credential is acquired and released inside pqsecure_open_gss(), as before. GSS_C_DELEG_FLAG is set there only when gssdelegation=1.
One limit: this server has gss_accept_delegation = off, so pg_stat_gssapi.credentials_delegated is false with or without the patch. I could confirm that libpq requests delegation, with a forwardable TGT, but not that the server receives the credentials.
Thanks for back-patching to 16. I'm happy to test the back-branch versions too.
Matt
Matt Carter <Matt.Carter@twosigma.com> writes:
Tom Lane wrote:
So I think what we really need is more like the attached. This passes
our own test suite, but I wonder if you can exercise it in your own
environment.
v1 works here.
Great, that gives me enough confidence to proceed. Thanks for
following through on this problem!
regards, tom lane
Import Notes
Reply to msg id not found: BL3PR08MB7283F6C6651ACC529F00CCDC996D2@BL3PR08MB7283.namprd08.prod.outlook.com
On Fri, Oct 30, 2026 at 9:55 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Thanks for testing that, but after looking closer, I believe that
pg_GSS_continue has even more problems here. It will attempt to
acquire credentials whether or not GSS delegation is enabled,
contrary to the adjacent comment and unlike what happens in
pqsecure_open_gss. And if GSS delegation is enabled, it will
set the GSS_C_DELEG_FLAG whether or not it was able to acquire
credentials, which seems not very sane and again different from
the logic in pqsecure_open_gss.
Hi! I don't have any more time today to dig, but I took a partial look.
+1 for the name change to pg_GSS_get_cred_cache().
/* Only try to acquire credentials if GSS delegation isn't disabled. */ - if (!pg_GSS_have_cred_cache(&conn->gcred)) - conn->gcred = GSS_C_NO_CREDENTIAL; - if (conn->gssdelegation && conn->gssdelegation[0] == '1') - gss_flags |= GSS_C_DELEG_FLAG; + { + if (conn->gcred == GSS_C_NO_CREDENTIAL) + conn->gcred = pg_GSS_get_cred_cache(); + if (conn->gcred != GSS_C_NO_CREDENTIAL) + gss_flags |= GSS_C_DELEG_FLAG; + }
This might be equivalent logic, but I'm not sure. The current code
always requests delegation if conn->gssdelegation is true (so if
pg_GSS_get_cred_cache()/gss_acquire_cred() fail, then delegation will
still be attempted with GSS_C_NO_CREDENTIAL). The new code requests
delegation only if gss_acquire_cred() succeeds -- if that fails but
the later GSS_C_NO_CREDENTIAL attempt succeeds for some weird
GSS-specific reason, shouldn't we still try to delegate as directed?
(Maybe if gss_acquire_cred() fails, the later GSS_C_NO_CREDENTIAL
attempt is also guaranteed to fail? But if that's true then this code
should look a lot different IMO.)
Separately from that:
+ if (conn->gcred == GSS_C_NO_CREDENTIAL) + conn->gcred = pg_GSS_get_cred_cache();
Are there any delegation cases where we were actually expecting to
overwrite the previous, valid, credentials? I deleted
pg_GSS_continue's pg_GSS_have_cred_cache() call entirely, and the
tests still passed on my machine, which makes me worried that maybe
there's an edge case that was never recorded. (Alternatively, maybe
the call is completely unnecessary.)
Thanks,
--Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
On Fri, Oct 30, 2026 at 9:55 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
/* Only try to acquire credentials if GSS delegation isn't disabled. */ - if (!pg_GSS_have_cred_cache(&conn->gcred)) - conn->gcred = GSS_C_NO_CREDENTIAL; - if (conn->gssdelegation && conn->gssdelegation[0] == '1') - gss_flags |= GSS_C_DELEG_FLAG; + { + if (conn->gcred == GSS_C_NO_CREDENTIAL) + conn->gcred = pg_GSS_get_cred_cache(); + if (conn->gcred != GSS_C_NO_CREDENTIAL) + gss_flags |= GSS_C_DELEG_FLAG; + }
This might be equivalent logic, but I'm not sure.
You're right, it's intentionally not equivalent logic.
The current code
always requests delegation if conn->gssdelegation is true (so if
pg_GSS_get_cred_cache()/gss_acquire_cred() fail, then delegation will
still be attempted with GSS_C_NO_CREDENTIAL). The new code requests
delegation only if gss_acquire_cred() succeeds -- if that fails but
the later GSS_C_NO_CREDENTIAL attempt succeeds for some weird
GSS-specific reason, shouldn't we still try to delegate as directed?
How can delegation succeed if there's nothing to delegate? Also,
the equivalent stanza in pqsecure_open_gss (fe-secure-gssapi.c,
lines 641-656 in master) behaves this way. I'm much more prepared
to believe that pqsecure_open_gss's behavior is intentional; for
one thing because it matches its comments, which pg_GSS_continue
didn't.
Are there any delegation cases where we were actually expecting to
overwrite the previous, valid, credentials?
Doubt it. There is nothing different about what a repeat call
to pg_GSS_get_cred_cache will do, so I'd expect it to fetch
credentials that are exactly like the old ones, modulo perhaps
a slightly newer expiration time.
I deleted
pg_GSS_continue's pg_GSS_have_cred_cache() call entirely, and the
tests still passed on my machine, which makes me worried that maybe
there's an edge case that was never recorded. (Alternatively, maybe
the call is completely unnecessary.)
I wondered about that too. I tried running the src/test/kerberos
tests with --enable-coverage, and found that they reach every call
of pg_GSS_get_cred_cache *except* the one in pqsecure_open_gss ---
but that's because we already have credentials there, as evidenced by
the fact that the "gss_flags |= GSS_C_DELEG_FLAG" line is reached.
So we do seem to need pg_GSS_continue's call.
Also, if you look at 6633cfb21, this whole stanza in pg_GSS_continue
was introduced then; before that we never passed credentials nor
GSS_C_DELEG_FLAG to that call of gss_init_sec_context, or the other
one either. So I have to think that's the intended behavior in
non-delegation mode. In any case, it does not seem to make sense
to be passing different parameters to successive calls of
gss_init_sec_context.
But I'm mostly unfamiliar with the details of GSS, so it's possible
I'm all wet here.
regards, tom lane
On Fri, Oct 30, 2026 at 12:32 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
The current code
always requests delegation if conn->gssdelegation is true (so if
pg_GSS_get_cred_cache()/gss_acquire_cred() fail, then delegation will
still be attempted with GSS_C_NO_CREDENTIAL). The new code requests
delegation only if gss_acquire_cred() succeeds -- if that fails but
the later GSS_C_NO_CREDENTIAL attempt succeeds for some weird
GSS-specific reason, shouldn't we still try to delegate as directed?How can delegation succeed if there's nothing to delegate?
IIUC, GSS_C_NO_CREDENTIAL doesn't actually mean "no credentials" to
gss_init_sec_context(), it means "get the default credentials from
somewhere magic." I don't know enough about the API to know whether
the gss_acquire_cred() call we're making separately is equivalent to
that magic behavior.
Also,
the equivalent stanza in pqsecure_open_gss (fe-secure-gssapi.c,
lines 641-656 in master) behaves this way. I'm much more prepared
to believe that pqsecure_open_gss's behavior is intentional; for
one thing because it matches its comments, which pg_GSS_continue
didn't.
Hmm, okay.
Are there any delegation cases where we were actually expecting to
overwrite the previous, valid, credentials?Doubt it. There is nothing different about what a repeat call
to pg_GSS_get_cred_cache will do, so I'd expect it to fetch
credentials that are exactly like the old ones, modulo perhaps
a slightly newer expiration time.
I was worried about cases like pg_store_delegated_credential(), where
we manipulate the default creds in memory. But if we never do anything
like that in between client calls to pg_GSS_get_cred_cache(), then
it's probably not a concern.
I tried running the src/test/kerberos
tests with --enable-coverage, and found that they reach every call
of pg_GSS_get_cred_cache *except* the one in pqsecure_open_gss ---
but that's because we already have credentials there, as evidenced by
the fact that the "gss_flags |= GSS_C_DELEG_FLAG" line is reached.
So we do seem to need pg_GSS_continue's call.
Huh. I'll try to take a closer look when I can.
In any case, it does not seem to make sense
to be passing different parameters to successive calls of
gss_init_sec_context.
Agreed [1]https://www.rfc-editor.org/info/rfc7546/#section-3:
The initiator should not change any of the input
parameters to GSS_Init_sec_context() between calls to it during the
loop, with the exception of the input_token parameter, which will
contain a message from the acceptor after the initial call, and the
input_context_handle, which must be the result returned in the
output_context_handle of the previous call to GSS_Init_sec_context()
(GSS_C_NO_CONTEXT for the first call). RFC 2743 only
requires that the claimant_cred_handle argument remain constant over
all calls in the loop, but the other non-excepted arguments should
also remain fixed for reliable operation.
Thanks,
--Jacob
Jacob Champion <jacob.champion@enterprisedb.com> writes:
On Fri, Oct 30, 2026 at 12:32 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
How can delegation succeed if there's nothing to delegate?
IIUC, GSS_C_NO_CREDENTIAL doesn't actually mean "no credentials" to
gss_init_sec_context(), it means "get the default credentials from
somewhere magic." I don't know enough about the API to know whether
the gss_acquire_cred() call we're making separately is equivalent to
that magic behavior.
Oh, that's very interesting!
If you look back at v14, there is just one call to
pg_GSS_have_cred_cache, and AFAICS it happens unconditionally if
we attempt GSS encryption, and we don't reach any other
GSS code if it fails.
In the code as it stands, my coverage result says it's possible to
reach pg_GSS_continue without having already obtained credentials, but
I don't quite see how. select_next_encryption_method doesn't look
like it will permit using ENC_GSSAPI unless we obtain credentials.
Too bad this code is so poorly documented. It's really pretty unclear
what the intent is.
regards, tom lane