BUG #19636: heap_fill_tuple misses short varlena conversion after EOH_flatten_into, causing 3-byte waste per row

пт, 21 авг. 2026 г. в 17:44, PG Bug reporting form <noreply@postgresql.org>:

The following bug has been logged on the website:

Bug reference: 19636
Logged by: anliuan
Email address: 17801022106@163.com
PostgreSQL version: 19beta1
Operating system: centos
Description:

When a PL/pgSQL function/procedure inserts an array value that arrives as
an
expanded object (via EOH_flatten_into), the resulting tuple stores the
varlena with a 4-byte header instead of converting it to a
1-byte short varlena header, wasting 3 bytes per row for arrays <= 127
bytes total size.

Plain INSERT correctly uses the short varlena format; only the PL/pgSQL
expanded-object path is affected.

Steps to reproduce:

CREATE TABLE tt2(a text[]);
INSERT INTO tt2 VALUES('{aaaaa}'); -- plain INSERT

CREATE OR REPLACE FUNCTION call_insert_acl(racl text[]) RETURNS int AS $$
BEGIN
INSERT INTO tt2 VALUES(racl);
RETURN 1;
END;
$$ LANGUAGE plpgsql;

SELECT call_insert_acl('{aaaaa}'); -- PL/pgSQL INSERT

SELECT a, pg_column_size(a) FROM tt2;

Expected result: Both rows should have the same pg_column_size (33 bytes —
short varlena with 1-byte header).

Actual result:

┌─────────────────┬────────────────┬───────────────────────────────┐
│ row │ pg_column_size │ header format │
├─────────────────┼────────────────┼───────────────────────────────┤
│ plain INSERT │ 33 │ short varlena (1-byte header) │
├─────────────────┼────────────────┼───────────────────────────────┤
│ PL/pgSQL INSERT │ 36 │ 4-byte header │
└─────────────────┴────────────────┴───────────────────────────────┘
Root cause:

In src/backend/access/common/heaptuple.c, heap_fill_tuple has four
varlena
paths. The EOH_flatten_into path (for expanded objects) produces a
4-byte-header varlena and writes it directly, without checking
VARATT_CAN_MAKE_SHORT:

/* Current code — line ~229 */
if (VARATT_IS_EXTERNAL_EXPANDED(val))
{
ExpandedObjectHeader *eoh = DatumGetEOHP(values[i]);
data = (char *) ATT_ALIGN_NOMINAL(data, att[i]->attalign);
data_length = EOH_get_flat_size(eoh);
EOH_flatten_into(eoh, data, data_length);
/* BUG: no VARATT_CAN_MAKE_SHORT check here */
}

The normal 4-byte-header path correctly performs this check:

/* Normal path — line ~240 */
else if (VARLENA_ATT_IS_PACKABLE(att[i]) && VARATT_CAN_MAKE_SHORT(val))
{
data_length = VARATT_CONVERTED_SHORT_SIZE(val);
SET_VARSIZE_SHORT(data, data_length);
}

heap_compute_data_size has the same omission — it uses
EOH_get_flat_size()
without considering the short-header size reduction, so the size
calculation
and fill logic are consistently wrong (no memory
overrun, just wasted space).

History
-------
Short headers date from 3e23b68dac0 (2007, "Support varlena fields with
single-byte headers and unaligned storage"). heap_form_tuple has packed
inline 4-byte-header varlenas via VARATT_CAN_MAKE_SHORT ever since the
modern heap_form_tuple path.

Expanded objects arrived in 1dc5ebc9077 (2015, "Support expanded
objects, particularly arrays, for better performance"). That commit
added an EXTERNAL_EXPANDED branch in heap_compute_data_size / fill_val
that flattens with EOH_flatten_into and writes the result as-is.
Flatteners are required to produce an inline 4-byte-header varlena
(see expandeddatum.h). The new branch never applied the existing
short-header conversion afterward. So this looks like an omission from
day one of expanded objects, not a later regression.

Proposal Fix
---
In those EXTERNAL_EXPANDED arms, if the attribute is packable and the
flat size from EOH_get_flat_size() would fit a short header, flatten
into a temporary palloc buffer and emit the short form into the tuple.
Otherwise keep the previous path (align and flatten straight into the
tuple).

VARATT_CAN_MAKE_SHORT cannot be used on the expanded toast pointer
itself. It requires a 4B_U varlena. Before flattening we only have
the flat size, so the patch uses a size-only helper matching that
macro's length rule.

The temporary buffer is needed because EOH_flatten_into expects a
maxaligned destination (same constraint as datumSerialize), while short
packing writes at an unaligned data pointer. The temp is not pfree'd.
fill_val can run with CurrentMemoryContext as a BumpContext. On
master / REL_19, RecursiveUnion UNION DISTINCT stores hashed tuples in
a BumpContext tuplescxt (c106ef08071, "Use BumpContext contexts in
TupleHashTables"). LookupTupleHashEntry switches to that context
before ExecCopySlotMinimalTupleExtra → heap_form_minimal_tuple. Bump
does not support pfree. A variant that pfree'd the temp failed the
"with" regress test on 19/master with:

ERROR: pfree is not supported by the bump memory allocator

REL_18 still used AllocSet for that context, so the same pfree passed
there. The short-packable flat size is at most about 130 bytes. The
chunk is reclaimed when the context is reset.

Alternatives considered: Stack buffer was considered (fits the size bound),
but palloc with EOH_flatten_into seemed better to me.

--
Regards,
Rachitskiy Andrey

пт, 21 авг. 2026 г. в 19:37, Andrey Rachitskiy <pl0h0yp1@gmail.com>:

History
-------
Short headers date from 3e23b68dac0 (2007, "Support varlena fields with
single-byte headers and unaligned storage"). heap_form_tuple has packed
inline 4-byte-header varlenas via VARATT_CAN_MAKE_SHORT ever since the
modern heap_form_tuple path.

Expanded objects arrived in 1dc5ebc9077 (2015, "Support expanded
objects, particularly arrays, for better performance"). That commit
added an EXTERNAL_EXPANDED branch in heap_compute_data_size / fill_val
that flattens with EOH_flatten_into and writes the result as-is.
Flatteners are required to produce an inline 4-byte-header varlena
(see expandeddatum.h). The new branch never applied the existing
short-header conversion afterward. So this looks like an omission from
day one of expanded objects, not a later regression.

Proposal Fix
---
In those EXTERNAL_EXPANDED arms, if the attribute is packable and the
flat size from EOH_get_flat_size() would fit a short header, flatten
into a temporary palloc buffer and emit the short form into the tuple.
Otherwise keep the previous path (align and flatten straight into the
tuple).

VARATT_CAN_MAKE_SHORT cannot be used on the expanded toast pointer
itself. It requires a 4B_U varlena. Before flattening we only have
the flat size, so the patch uses a size-only helper matching that
macro's length rule.

The temporary buffer is needed because EOH_flatten_into expects a
maxaligned destination (same constraint as datumSerialize), while short
packing writes at an unaligned data pointer. The temp is not pfree'd.
fill_val can run with CurrentMemoryContext as a BumpContext. On
master / REL_19, RecursiveUnion UNION DISTINCT stores hashed tuples in
a BumpContext tuplescxt (c106ef08071, "Use BumpContext contexts in
TupleHashTables"). LookupTupleHashEntry switches to that context
before ExecCopySlotMinimalTupleExtra → heap_form_minimal_tuple. Bump
does not support pfree. A variant that pfree'd the temp failed the
"with" regress test on 19/master with:

ERROR: pfree is not supported by the bump memory allocator

REL_18 still used AllocSet for that context, so the same pfree passed
there. The short-packable flat size is at most about 130 bytes. The
chunk is reclaimed when the context is reset.

Alternatives considered: Stack buffer was considered (fits the size
bound), but palloc with EOH_flatten_into seemed better to me.

A follow-up on the repro and on the regress in v1.

The heaptuple omission is real, but the reporter's one-shot example often
does not reach it. With the default plan_cache_mode = auto, the first
executions of the PL/pgSQL INSERT use a custom plan. The bound array
parameter is substituted during planning. datumCopy flattens the
expanded object there, so fill_val later sees an ordinary 4-byte-header
varlena and packs it to short as usual. Both rows then show
pg_column_size 33 even without the fix.

I checked both cases with gdb on an unpatched build, breaking on
EOH_flatten_into.

1. Default plan_cache_mode (auto), one-shot PL/pgSQL INSERT — custom plan

EOH_flatten_into is called once, with allocated_size 36, from datumCopy
during planning (parameter substitution in eval_const_expressions):

#0 EOH_flatten_into (... allocated_size=36)
#1 datumCopy
#3 eval_const_expressions_mutator
...
# BuildCachedPlan / GetCachedPlan
# SPI_execute_plan_with_paramlist
# plpgsql exec_stmt_execsql

There is no fill_val frame on that hit. heap_form_tuple later sees an
ordinary flat 4-byte-header varlena and applies VARATT_CAN_MAKE_SHORT as
usual. pg_column_size is 33 for the plain INSERT and 33 for the
PL/pgSQL INSERT.

2. Same SQL with a generic plan

SET plan_cache_mode = force_generic_plan;

EOH_flatten_into is again called with allocated_size 36, but the caller
is the EXTERNAL_EXPANDED arm in fill_val:

#0 EOH_flatten_into (... allocated_size=36)
#1 fill_val (... heaptuple.c)
#2 heap_fill_tuple
#3 heap_form_tuple
...
# ExecModifyTable / SPI / plpgsql

That arm writes the flattener output as-is, so the PL/pgSQL row keeps
the 4-byte header. Without the fix pg_column_size is 33 (plain INSERT)
and 36 (PL/pgSQL). With the fix both are 33.

Under plan_cache_mode = auto the same fill_val path appears after the
plan cache switches from custom to generic (after several executions of
the INSERT). force_generic_plan just makes that path reliable on the
first call.

v2 of the patch is attached. The heaptuple change is the same as v1.
The plpgsql regress now sets force_generic_plan around the test so case
2 is what we cover. Case 1 would pass even without the fix.

--
Regards,
Rachitskiy Andrey