Memory leak in statext_ndistinct_build() during ANALYZE
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
This thread has been committed, so CI has stopped here. Anything below is the last result it produced.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t254113psql -h localhost -U postgresBuilt from patchset v1 (message #1), October 13, 2026 at 09:01 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t254113_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t254113_1 && git checkout t254113_1Patchset v1 (message #1) is on t254113_1
Hi hackers,
While testing ANALYZE with large statistics targets, I found that
building ndistinct extended statistics leaks memory proportional to the
sample size for every combination of columns.
Example:
postgresql.conf settings
```
default_statistics_target = 1000
```
Commands
```
SET client_min_messages = log;
SET log_statement_stats = on;
DROP TABLE IF EXISTS ndist_leak;
CREATE TABLE ndist_leak AS
SELECT (random() * 3000)::int AS a,
(random() * 3000)::int AS b,
(random() * 50)::int AS c,
(random() * 20)::int AS d,
(random() * 10)::int AS e,
(random() * 5)::int AS f
FROM generate_series(1, 1000000);
CREATE STATISTICS ndist_leak_nd (ndistinct) ON a, b, c, d, e, f FROM
ndist_leak;
ANALYZE ndist_leak;
```
*Peak memory of the backend:*
Before patch - 971076 kB max resident size
After patch - 112780 kB max resident size
ndistinct_for_combination() allocates the items, values and isnull
arrays for the whole sample, plus the MultiSortSupport, and never frees
them. statext_ndistinct_build() calls it once per combination of
columns, so these allocations accumulate until the whole ndistinct
object is built.
The attached patch changes statext_ndistinct_build() to call
ndistinct_for_combination() in a separate memory context and reset it
after each combination, the same way as is already done for
dependency_degree() in statext_dependencies_build().
Similar memory issues in extended statistics were fixed in [0]/messages/by-id/20210915200928.GP831@telsasoft.com. The
original report there actually used ndistinct objects, but the fixes
addressed memory accumulated across statistics objects and in
dependency_degree(); the per-combination allocations in
ndistinct_for_combination() were not discussed there.
Any suggestions?
[0]: /messages/by-id/20210915200928.GP831@telsasoft.com
/messages/by-id/20210915200928.GP831@telsasoft.com
--
Best regards,
Ilia Evdokimov,
Tantor Labs LLC,
https://tantorlabs.com/
On Fri, Oct 09, 2026 at 12:06:20AM +0500, Ilia Evdokimov wrote:
ndistinct_for_combination() allocates the items, values and isnull arrays
for the whole sample, plus the MultiSortSupport, and never frees them.
statext_ndistinct_build() calls it once per combination of columns, so these
allocations accumulate until the whole ndistinct object is built.
Nice catch. Indeed, that's annoying.
The attached patch changes statext_ndistinct_build() to call
ndistinct_for_combination() in a separate memory context and reset it after
each combination, the same way as is already done for dependency_degree() in
statext_dependencies_build().
Bumping default_statistics_target to 10000 and running ANALYZE in a
\watch loop, it looks like your patch is keeping the memory used under
check.
Similar memory issues in extended statistics were fixed in [0]. The original
report there actually used ndistinct objects, but the fixes addressed memory
accumulated across statistics objects and in dependency_degree(); the
per-combination allocations in ndistinct_for_combination() were not
discussed there.
This part refers to 83772cc78e03 and ad8a166ca868 (ndistinct with
memory context reset as you are mentioning), and both got backpatched.
It sounds to me that we should do the same here, down to v14.
What you have looks OK at quick glance. I'll probably process that
next week after v19 RC1 is tagged, unless there is an objection of any
kind.
Thanks for the report and the patch!
--
Michael
On Fri, Oct 09, 2026 at 03:45:18PM +0900, Michael Paquier wrote:
What you have looks OK at quick glance. I'll probably process that
next week after v19 RC1 is tagged, unless there is an objection of any
kind.
The RC1 tag has been pushed, so applied this one.
--
Michael