Limitation relates to memory allocation

On 14.10.24 08:03, Ekaterina Kiryanova wrote:

We encountered an issue related to internal memory allocation limit:
ERROR: invalid memory alloc request size

In the documentation on limits: https://www.postgresql.org/docs/17/
limits.html
the description suggests that only a single field is limited to 1GB,
which could imply that the total tuple size can be larger. So as we
planned to store three columns of 1GB each in a table and attempted to
insert this data, we got the error.

Our research showed that the limit is imposed by the palloc() function,
regardless of whether it is a tuple or not, and if the data is
serialized or dumped, the effective limit can be even lower, typically
around 512MB per row. So for allocations exceeding 1GB, the
palloc_extended() function can be used. Please correct me if I'm wrong.

I prepared a small patch for master, if it's worth clarifying, could you
please review the attachment?

The 1 GB limit in palloc() is a safety check, when the code shouldn't be
allocating more than that. Code that legitimately wants to allocate
more than 1 GB can use the MCXT_ALLOC_HUGE flag.

If you see this error, then that could either be corruption somewhere
(the kind of thing this safety check is meant to catch) or the code is
buggy.

In either case, I don't know that it is appropriate to document this as
an externally visible system limitation.

On Mon, 14 Oct 2024 at 19:03, Ekaterina Kiryanova
<e.kiryanova@postgrespro.ru> wrote:

Our research showed that the limit is imposed by the palloc() function,
regardless of whether it is a tuple or not, and if the data is
serialized or dumped, the effective limit can be even lower, typically
around 512MB per row. So for allocations exceeding 1GB, the
palloc_extended() function can be used. Please correct me if I'm wrong.

I think it would be nice to document the row length limitation and
also add a caveat to the "field size" row to mention that outputting
bytea columns larger than 512MB can be problematic and storing values
that size or above is best avoided.

I don't think wording like: "The practical limit is less than 1 GB" is
going to be good enough as it's just not specific enough. The other
places that talk about practical limits on that page are mostly there
because it's likely impossible that anyone could actually reach the
actual limit. For example, 2^32 databases is likely a limit that
nobody would be able to get close. It's pretty easy to hit the bytea
limit, however:

postgres=# create table b (a bytea);
CREATE TABLE
Time: 2.634 ms
postgres=# insert into b values(repeat('a',600*1024*1024)::bytea);
INSERT 0 1
Time: 9725.320 ms (00:09.725)
postgres=# \o out.txt
postgres=# select * from b;
ERROR: invalid memory alloc request size 1258291203
Time: 209.082 ms

that took me about 10 seconds, so I disagree storing larger bytea
values is impractical.

David