Invalid memory alloc querying field type bytea

"Thompson, Shon" <Shon.Thompson@MaineGeneral.org> writes:

I'm getting errors when I query a table that uses data type bytea. I can query the table if I don't include the field for data type bytea. Here are the two error messages I get. The first error is one generated from visual studio:
"ERROR: XX000: invalid memory alloc request size 1294068297"

Looks like a corrupt-data problem. I doubt it's directly the fault of
bytea; any variable-length field would have given a similar type of error,
because what this seems to be is a damaged field length word.

Do you see any clearly bogus data in other columns of the table? That
might help isolate which row(s) have gotten damaged. It seems somewhat
unlikely that only the bytea field header would be clobbered and not
any adjacent data.

The questions you should be asking yourself include whether your PG
version is up-to-date and whether the storage stack is reliable.

regards, tom lane

I tried installing the latest version of postgress, and I get the same error. Postgres returns the smaller rows without errors, but cannot return the two largest bytea fields. The length function for the bytea data of these two rows returns:
647,034,150
819,319,303

This is a windows machine. I'd expect the OS to use paging file if there's not enough memory. My computer doesn't seem to be memory bound because I have 16 gig and 6 gig free when the query runs. I'm getting the same error on these two versions of postgres:
11.2 build 1914
12.3 build 1914

Shon Thompson
Office:  207-624-4391

-----Original Message-----
From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Wednesday, June 17, 2020 2:46 PM
To: Thompson, Shon <Shon.Thompson@MaineGeneral.org>
Cc: pgsql-bugs@lists.postgresql.org
Subject: <EXTERNAL> Re: Invalid memory alloc querying field type bytea

CAUTION: EXTERNAL EMAIL --- THIS EMAIL ORIGINATED OUTSIDE OF MAINEGENERAL. DO NOT CLICK ANY LINKS OR OPEN ANY ATTACHMENTS UNLESS YOU KNOW THE SENDER. REPORT ALL SUSPICIOUS EMAIL TO SPAM@MAINEGENERAL.ORG <mailto:SPAM@MAINEGENERAL.ORG>

"Thompson, Shon" <Shon.Thompson@MaineGeneral.org> writes:

I'm getting errors when I query a table that uses data type bytea. I can query the table if I don't include the field for data type bytea. Here are the two error messages I get. The first error is one generated from visual studio:
"ERROR: XX000: invalid memory alloc request size 1294068297"

Looks like a corrupt-data problem. I doubt it's directly the fault of bytea; any variable-length field would have given a similar type of error, because what this seems to be is a damaged field length word.

Do you see any clearly bogus data in other columns of the table? That might help isolate which row(s) have gotten damaged. It seems somewhat unlikely that only the bytea field header would be clobbered and not any adjacent data.

The questions you should be asking yourself include whether your PG version is up-to-date and whether the storage stack is reliable.

regards, tom lane

On 7/8/20 11:25 AM, Thompson, Shon wrote:

Postgres returns the smaller rows without errors, but cannot return the two
largest bytea fields. The length function for the bytea data of these two
rows returns:

647,034,150
819,319,303

"ERROR: XX000: invalid memory alloc request size 1294068297"

The problem is probably byteaout rendering the bytes as (presumably) hex, which
takes ~twice as many bytes as the data, and runs afoul of MaxAllocSize.

Joe

--
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development