BUG #19857: Stats restore rejects empty arrays, losing range and array stats

Started by PG Bug reporting form13 days ago13 messagesbugs
Jump to latest
#1PG Bug reporting form
noreply@postgresql.org

The following bug has been logged on the website:

Bug reference: 19857
Logged by: Atish Prakash
Email address: 125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system: Linux x86_64
Description:

If a range column has fewer than two non-empty values, ANALYZE stores an
empty
range length histogram. Restoring those statistics fails with a WARNING, so
range_empty_frac is lost too, and estimates change a lot.

CREATE TABLE r(a int4range);
INSERT INTO r SELECT 'empty' FROM generate_series(1, 10000);
ANALYZE r;

Then dump with "pg_dump --statistics" and restore into an empty database:

master: WARNING: "range_length_histogram" must be a one-dimensional
array
18.6: WARNING: argument "range_length_histogram" must be specified
when
argument "range_empty_frac" is specified (message shortened)

Estimates on master, with autovacuum disabled on the table so that a later
ANALYZE
does not replace the statistics:

query actual before after
SELECT * FROM r WHERE a <@ '[1,2)' 10000 10000 89
SELECT * FROM r WHERE a && '[1,2)' 0 1 178

A table with one non-empty value and 9,999 empty ones is affected the same
way
(estimate 9999 before, 87 after, for the first form of query, with actual
10000).
Multirange columns also produce the warning.

pg_upgrade loses them too: after an upgrade from 18.6 to 19rc1,
range_empty_frac for
table r went from 1 to NULL, and the first query above was estimated at 50
rows.

On 19rc1 and master the same check rejects most_common_elems when it is
empty, which
commit 261f89a976 makes common. An ordinary array column with all-distinct
elements
is enough, and tsvector columns are affected too:

CREATE TABLE tags(id int, a int[]);
INSERT INTO tags
SELECT i, ARRAY[i, i + 1000000] FROM generate_series(1, 50000) i;
ANALYZE tags;

WARNING: "most_common_elems" must be a one-dimensional array

query actual before after
SELECT * FROM tags WHERE a @> '{5}' 1 3 ~270
SELECT * FROM tags WHERE a && '{5,6}' 2 7 ~530

Extended statistics are hit harder on master. An empty array in one
expression's
statistics rejects the statistics for every expression in that object,
including
unrelated ones:

WARNING: could not parse "exprs": invalid element in expression -1

Expected: the statistics restore as they were, without a warning.

Cause: on master, pg_stats shows range_length_histogram as '{}' and pg_dump
emits
that. statatt_build_stavalues() requires a one-dimensional array, and an
empty
array has zero dimensions. On 18.6, pg_stats shows it as NULL, so pg_dump
omits it
while still emitting range_empty_frac, and the restore function rejects that
pair.

I have not written a patch.

Found with dump and restore differential tests. I reproduced and checked the
result myself.

#2Corey Huinker
corey.huinker@gmail.com
In reply to: PG Bug reporting form (#1)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Tue, Nov 10, 2026 at 4:48 AM PG Bug reporting form <
noreply@postgresql.org> wrote:

The following bug has been logged on the website:

Bug reference: 19857
Logged by: Atish Prakash
Email address: 125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system: Linux x86_64
Description:

If a range column has fewer than two non-empty values, ANALYZE stores an
empty
range length histogram. Restoring those statistics fails with a WARNING, so
range_empty_frac is lost too, and estimates change a lot.

CREATE TABLE r(a int4range);
INSERT INTO r SELECT 'empty' FROM generate_series(1, 10000);
ANALYZE r;

Then dump with "pg_dump --statistics" and restore into an empty database:

master: WARNING: "range_length_histogram" must be a one-dimensional
array
18.6: WARNING: argument "range_length_histogram" must be specified
when
argument "range_empty_frac" is specified (message shortened)

Estimates on master, with autovacuum disabled on the table so that a later
ANALYZE
does not replace the statistics:

query actual before after
SELECT * FROM r WHERE a <@ '[1,2)' 10000 10000 89
SELECT * FROM r WHERE a && '[1,2)' 0 1 178

A table with one non-empty value and 9,999 empty ones is affected the same
way
(estimate 9999 before, 87 after, for the first form of query, with actual
10000).
Multirange columns also produce the warning.

pg_upgrade loses them too: after an upgrade from 18.6 to 19rc1,
range_empty_frac for
table r went from 1 to NULL, and the first query above was estimated at 50
rows.

On 19rc1 and master the same check rejects most_common_elems when it is
empty, which
commit 261f89a976 makes common. An ordinary array column with all-distinct
elements
is enough, and tsvector columns are affected too:

CREATE TABLE tags(id int, a int[]);
INSERT INTO tags
SELECT i, ARRAY[i, i + 1000000] FROM generate_series(1, 50000) i;
ANALYZE tags;

WARNING: "most_common_elems" must be a one-dimensional array

query actual before after
SELECT * FROM tags WHERE a @> '{5}' 1 3 ~270
SELECT * FROM tags WHERE a && '{5,6}' 2 7 ~530

Extended statistics are hit harder on master. An empty array in one
expression's
statistics rejects the statistics for every expression in that object,
including
unrelated ones:

WARNING: could not parse "exprs": invalid element in expression -1

Expected: the statistics restore as they were, without a warning.

Cause: on master, pg_stats shows range_length_histogram as '{}' and pg_dump
emits
that. statatt_build_stavalues() requires a one-dimensional array, and an
empty
array has zero dimensions. On 18.6, pg_stats shows it as NULL, so pg_dump
omits it
while still emitting range_empty_frac, and the restore function rejects
that
pair.

I have not written a patch.

Found with dump and restore differential tests. I reproduced and checked
the
result myself.

Looking into this, probably a duplicate of 19842

#3Corey Huinker
corey.huinker@gmail.com
In reply to: Corey Huinker (#2)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

Looking into this, probably a duplicate of 19842

It's definitely a duplicate of 19842, but because it would need to be
backported to 18, we'd want to handle this one first, and build
the 19842-stuff on top of that.

I have relaxed the test in stats_check_required_arg() to allow for empty
arrays, which itself creates a burden of checking the number of dimensions
of an array before asking for the number of vals in the 0th dimension,
which needs to be done twice in attribute_statistics_update_internal() and
nowhere else, mostly because there are only two arrays (most_common_values
and most common_freqs) that need to have an equal number of values (MCELEM
has additional values over its freqs, and range_length_hisotgram is paired
with the scalar range_null_frac as a way to save slots, I presume.

Added additional test cases to show that the length agreement requirement
still works on MCV/MCF when one is empty, and that both store empty arrays
without complaint.

Attachments:

t258389_3
v1-0001-Allow-empty-arrays-as-input-for-pg_restore_attrib.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Allow-empty-arrays-as-input-for-pg_restore_attrib.patchDownload+88-8
#4Corey Huinker
corey.huinker@gmail.com
In reply to: Corey Huinker (#3)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

Same patch, commit message updated to have both discussion threads.

Attachments:

t258389_4
v2-0001-Allow-empty-arrays-as-input-for-pg_restore_attrib.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Allow-empty-arrays-as-input-for-pg_restore_attrib.patchDownload+88-8
#5Michael Paquier
michael@paquier.xyz
In reply to: Corey Huinker (#4)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Tue, Nov 10, 2026 at 08:50:39PM -0500, Corey Huinker wrote:

Same patch, commit message updated to have both discussion threads.

arr = DatumGetArrayTypeP(PG_GETARG_DATUM(argnum));

- if (ARR_NDIM(arr) != 1)
+ if (ARR_NDIM(arr) > 1)
{
ereport(WARNING,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),

This change would trigger a CVE, I suspect.. Under valgrind, and for
example these sequences:
CREATE TABLE t (a int);
-- most_common_freqs
SELECT pg_restore_attribute_stats(
'schemaname', 'public', 'relname', 't', 'attname', 'a',
'inherited', false,
'most_common_vals', '{}'::text,
'most_common_freqs', '{}'::real[]);
EXPLAIN SELECT * FROM t WHERE a = 1;

CREATE TABLE t2 (b int[]);
-- most_common_elem_freqs
SELECT pg_restore_attribute_stats(
'schemaname', 'public', 'relname', 't2', 'attname', 'b',
'inherited', false,
'most_common_elems', '{}'::text,
'most_common_elem_freqs', '{}'::real[]);
EXPLAIN SELECT * FROM t2 WHERE b @> '{1}';
-- elem_count_histogram
SELECT pg_clear_attribute_stats('public', 't2', 'b', false);
SELECT pg_restore_attribute_stats(
'schemaname', 'public', 'relname', 't2', 'attname', 'b',
'inherited', false,
'elem_count_histogram', '{}'::real[]);
EXPLAIN SELECT * FROM t2, unnest(b);

Then get_attstatsslot() does an out-of-bound read for
most_common_freqs, most_common_elem_freqs and elem_count_histogram
because an array with 0 dimension has no follow-up data as far as I
can see, giving this pattern:
==804366== Invalid read of size 4
==804366== at 0x1A58924: get_attstatsslot (lsyscache.c:3636)
==804366== by 0x1950B6C: var_eq_const (selfuncs.c:422)
==804366== by 0x195063F: eqsel_internal (selfuncs.c:349)
==804366== by 0x1950297: eqsel (selfuncs.c:302)
==804366== by 0x1ACE474: FunctionCall4Coll (fmgr.c:1215)
==804366== by 0x1AD0C37: OidFunctionCall4Coll (fmgr.c:1451)
==804366== by 0x122CC7F: restriction_selectivity (plancat.c:2242)
==804366== by 0x1063476: clause_selectivity_ext (clausesel.c:854)
==804366== by 0x1060B84: clauselist_selectivity_ext (clausesel.c:136)
==804366== by 0x1060AB1: clauselist_selectivity (clausesel.c:106)
==804366== by 0x10823BD: set_baserel_size_estimates (costsize.c:5615)
==804366== by 0x104B64A: set_plain_rel_size (allpaths.c:641)

Instead of allowing all these arrays to be empty, we should snipe at
range_length_histogram *and* most_common_elems, and only all these
two without generalizing the rule. The other three are rejected
anyway at load, so blocking them at restore would be a consistency
move.

Perhaps we should update get_attstatsslot() as well to avoid the
over-read issue to begin with; that feels like a sloppy thing to do if
there is an empty array to process.

We'll likely need to do something similar for the extstats. One step
at a time. We could of course do a more relaxing approach like you
are proposing, but I cannot really get excited to let that go through
at restore time as the loads would fail even if get_attstatsslot() is
adjusted (actually this path should be adjusted anyway), and the
option-basis thing is not that invasive in the attached, at least it
seems so to me.

Attached is a cleaned-up patch. So, thoughts or comments?
--
Michael

Attachments:

t258389_5
0001-Allow-empty-arrays-of-values-in-pg_restore_attribute.patchtext/plain; charset=us-asciiDownload+123-10
#6Corey Huinker
corey.huinker@gmail.com
In reply to: Michael Paquier (#5)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Wed, Nov 11, 2026 at 2:41 AM Michael Paquier <michael@paquier.xyz> wrote:

On Tue, Nov 10, 2026 at 08:50:39PM -0500, Corey Huinker wrote:

Same patch, commit message updated to have both discussion threads.

arr = DatumGetArrayTypeP(PG_GETARG_DATUM(argnum));

- if (ARR_NDIM(arr) != 1)
+ if (ARR_NDIM(arr) > 1)
{
ereport(WARNING,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),

Nitpick: I think the error message itself needs to change based on
allow_empty, as true allows 1-D or empty, and false allows only 1-D.

Instead of allowing all these arrays to be empty, we should snipe at
range_length_histogram *and* most_common_elems, and only all these
two without generalizing the rule. The other three are rejected
anyway at load, so blocking them at restore would be a consistency
move.

I thought about that, and wanted to at least try this method first. I'm
glad we could rule it out early.

Perhaps we should update get_attstatsslot() as well to avoid the

over-read issue to begin with; that feels like a sloppy thing to do if
there is an empty array to process.

We'll likely need to do something similar for the extstats. One step
at a time. We could of course do a more relaxing approach like you
are proposing, but I cannot really get excited to let that go through
at restore time as the loads would fail even if get_attstatsslot() is
adjusted (actually this path should be adjusted anyway), and the
option-basis thing is not that invasive in the attached, at least it
seems so to me.

The more relaxing approach just means a lot more NDIM-checks, which undoes
the elegance of the initial fix.

This way seems not invasive at all, so I like it better.

Attached is a cleaned-up patch. So, thoughts or comments?

I'll redo the ext stats patchset to this pattern. Do you intend to commit
this soon? Asking so I know whether to rebase-away the 0001 patch.

#7Michael Paquier
michael@paquier.xyz
In reply to: Corey Huinker (#6)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Wed, Nov 11, 2026 at 08:46:56AM -0500, Corey Huinker wrote:

Nitpick: I think the error message itself needs to change based on
allow_empty, as true allows 1-D or empty, and false allows only 1-D.

Yeah, point taken that this could be a little bit better, in a shape
similar to what you are suggesting. I can see that we have similar
requirements in some of the array functions, though.. Will try to
shape that this way.

This way seems not invasive at all, so I like it better.

Okay, noted.

I'll redo the ext stats patchset to this pattern. Do you intend to commit
this soon? Asking so I know whether to rebase-away the 0001 patch.

Yeah, I intend to wrap this one this morning.
--
Michael

#8Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#7)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Thu, Nov 12, 2026 at 07:40:47AM +0900, Michael Paquier wrote:

Yeah, I intend to wrap this one this morning.

Applied this down to v19, to match with the empty array cases that
have been introduced in 261f89a976b.

I think that this leaves an extra case, though... In v17 and v18, it
is possible for range_empty_frac to be stored with
range_length_histogram as NULL. On v19 and HEAD,
range_length_histogram is an empty array, so the two are always set.
The restore functions require the two together when upgrading to 18~,
meaning that some stats values could be lost when upgrading/dumping
the stats from a v17 or v18.

That's one pattern that Atish was mentioning originally in his report,
even if I got a bit lost in my mind due to the dump/upgrade mentions
with v19 and HEAD.

I cannot really get excited to allow this combination in the restore
functions, though, as it means being more lenient on
range_length_histogram forever, and the v19~ decision of using an
empty array is much better to me in the long term. An extra argument
is that this case is super narrow anyway, unlikely to be hit in
practice? As far as I can see, you need a [multi]range column with
less than *2* non-empty values..

I agree that we should move on with fixing the extstats restore
functions, of course, in terms of the handling of the empty array
values, to map now with what the attribute restores are able to do.
And fix this second issue down to v19. Let's do that on the other
thread, we are done here for me.
--
Michael

#9Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#8)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Thu, Nov 12, 2026 at 10:26:00AM +0900, Michael Paquier wrote:

I cannot really get excited to allow this combination in the restore
functions, though, as it means being more lenient on
range_length_histogram forever, and the v19~ decision of using an
empty array is much better to me in the long term. An extra argument
is that this case is super narrow anyway, unlikely to be hit in
practice? As far as I can see, you need a [multi]range column with
less than *2* non-empty values..

Or we could do something on the dump side, forcing an empty array out
if seeing a NULL range_length_histogram.. Still not sure if that's
worth bothering, just saying.
--
Michael

#10Corey Huinker
corey.huinker@gmail.com
In reply to: Michael Paquier (#9)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Wed, Nov 11, 2026 at 8:32 PM Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Nov 12, 2026 at 10:26:00AM +0900, Michael Paquier wrote:

I cannot really get excited to allow this combination in the restore
functions, though, as it means being more lenient on
range_length_histogram forever, and the v19~ decision of using an
empty array is much better to me in the long term. An extra argument
is that this case is super narrow anyway, unlikely to be hit in
practice? As far as I can see, you need a [multi]range column with
less than *2* non-empty values..

Or we could do something on the dump side, forcing an empty array out
if seeing a NULL range_length_histogram.. Still not sure if that's
worth bothering, just saying.
--
Michael

I had also considered fixing it dump-side as well, which seems cleaner but
also means that the bug would persist in pre-existing dump files, though as
you mention it may not be worth bothering because this is a corner case
that only occurs if the histogram was less than 2 elements long, thus the
query plan wouldn't be that far off. Also, such a pre-existing dump file
would itself be restored to a cold-start database, and therefore unlikely
to be getting flooded with queries before vacuumdb has had time to run in
the way that a pg_upgrade-ed database would.

Commit looks good on master and REL_19_STABLE.

#11Corey Huinker
corey.huinker@gmail.com
In reply to: Corey Huinker (#10)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

Commit looks good on master and REL_19_STABLE.

But the problem still exists on 18, and that isn't a clean backport.
Michael, do you want me to take that one?

#12Michael Paquier
michael@paquier.xyz
In reply to: Corey Huinker (#11)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Wed, Nov 11, 2026 at 11:55:08PM -0500, Corey Huinker wrote:

But the problem still exists on 18, and that isn't a clean backport.
Michael, do you want me to take that one?

The only problem that can be reached is if we dump or upgrade from a
v17 or v18 instance with the pattern of a range_empty_frac set with
range_length_histogram as NULL, where we could perhaps switch
range_length_histogram to '{}' under-the-hood on the dump side if a
range_empty_frac exists. I don't want the restore functions to live
eternally with a path authorizing for range_length_histogram=NULL if
there is a range_empty_frac, especially now that
range_length_histogram has to be an empty array since v19~.

And the end, I have no plans to bother based on the table definition
this requires. I could still care with something on the dump side,
but I doubt that's worth the effort anyway. What's been done on HEAD
and v19 is enough for these new empty-array cases. :)
--
Michael

#13Corey Huinker
corey.huinker@gmail.com
In reply to: Michael Paquier (#12)
Re: BUG #19857: Stats restore rejects empty arrays, losing range and array stats

On Thu, Nov 12, 2026 at 12:10 AM Michael Paquier <michael@paquier.xyz>
wrote:

On Wed, Nov 11, 2026 at 11:55:08PM -0500, Corey Huinker wrote:

But the problem still exists on 18, and that isn't a clean backport.
Michael, do you want me to take that one?

The only problem that can be reached is if we dump or upgrade from a
v17 or v18 instance with the pattern of a range_empty_frac set with
range_length_histogram as NULL, where we could perhaps switch
range_length_histogram to '{}' under-the-hood on the dump side if a
range_empty_frac exists. I don't want the restore functions to live
eternally with a path authorizing for range_length_histogram=NULL if
there is a range_empty_frac, especially now that
range_length_histogram has to be an empty array since v19~.

And the end, I have no plans to bother based on the table definition
this requires. I could still care with something on the dump side,
but I doubt that's worth the effort anyway. What's been done on HEAD
and v19 is enough for these new empty-array cases. :)
--
Michael

Sorry, I should have clarified - I was considering "18 can't import
{}+empty_frac" a separate over the already-settled-not-gonna-do-it "18
can't import NULL+empty_frac" issue, but your response above indicates even
less appetite for allowing what would have to be synthetic stats just
because that's the future valid value.

I'll make the v18-specific pg_dump fix and additional patch in the other
thread.