compiler attribute alloc_size
In my ventures to apply more compiler attributes, I came to alloc_size.
This is to annotate memory allocation functions so that the compiler can
know the size of the allocated memory and so some cross-checking based
on that. This is all pretty straightforward and I think we should do it.
Andres had previously reported some issues he encountered in [0]/messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq. (That
message didn't go into details on what attributes or compiler options
were used, but it certainly looked very closely related.)
The issue is that under-allocating a struct is bad. With this
attribute, the compiler flags this, because it looks like a mistake.
For uses where this is legitimate, you'd normally use a struct with a
trailing flexible-array member, and we mostly make use of that. But
there are a couple of cases where we assign to a struct pointer less
memory than what the struct expects. In some cases, this is done with a
union where we only allocate enough memory for one branch of the union,
which seems like an intuitive thing to do but is actually also bad.
In the attached patches I have some fixes and/or workarounds for the
issues that I found.
Note that as documented this attribute and the associated warning
options are for diagnostics only, they don't change compilation. So in
that sense, once all warnings are fixed, this shouldn't create any more
hidden issues.
- varattrib_4b is a union of a normal varlena with a 4-byte header and 0
or more payload bytes, and a compressed format with a 4-byte header, a
4-byte "tcinfo", followed by payload data. The typical coding pattern
val = palloc(VARHDRSZ + len);
SET_VARSIZE(val, VARHDRSZ + len);
could get flagged by -Warray-bounds because the allocation could be too
short to cover both branches of the union. In this case, I think we
don't actually need the union, so I'm replacing this with two separate
structs.
- TRGM in contrib/pg_trgm/ is an instance of what [0]/messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq talks about: It
has a trailing flexible array member, but it also has trailing padding,
so allocating palloc(TRGMHDRSIZE) causes a warning, because it's less
than sizeof(TRGM). In my patch I work around this by allocating like
palloc(Max(TRGMHDRSIZE, sizeof(TRGM))).
- There is a bit of a maze of different structs and unions that underlie
the numeric type. The allocation (Numeric) palloc(NUMERIC_HDRSZ_SHORT)
gets flagged by the compiler. In my patch I change this to (Numeric)
palloc(sizeof(struct NumericData)), which allocates a bit more but makes
the sizes and types match. Maybe some more judicious reshuffling of the
underlying data structures could avoid this, or maybe it's okay like this.
- As of commit 6a3da13d376 from this week, FormData_pg_attribute has 4
bytes of trailing padding, and so sizeof(FormData_pg_attribute) is now
different from ATTRIBUTE_FIXED_PART_SIZE. This causes a warning in one
place. We had a discussion about this some time ago [1]/messages/by-id/7c7d41c1-fd5d-f9a5-afd5-f00168d91fe2@2ndquadrant.com, where the
opinions diverged between like, it's important to keep this distinction
in case we ever have padding bytes, and, it's already incorrect and
would be even more incorrect if we had padding bytes. ;-) Well, we're
here now, so perhaps this should be revisited.
This is all that I found. [0]/messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq lists a lot more sources of potential
problems, but apparently they are not flagged as they are currently used.
[0]: /messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq
/messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq
[1]: /messages/by-id/7c7d41c1-fd5d-f9a5-afd5-f00168d91fe2@2ndquadrant.com
/messages/by-id/7c7d41c1-fd5d-f9a5-afd5-f00168d91fe2@2ndquadrant.com
Attachments:
t259206_10001-Apply-alloc_size-attribute-to-memory-allocation-func.patchtext/plain; charset=UTF-8; name=0001-Apply-alloc_size-attribute-to-memory-allocation-func.patchDownload+56-40
0002-Fixes-for-alloc_size-attribute-varatt.patchtext/plain; charset=UTF-8; name=0002-Fixes-for-alloc_size-attribute-varatt.patchDownload+34-29
0003-Fixes-for-alloc_size-attribute-pg_trgm.patchtext/plain; charset=UTF-8; name=0003-Fixes-for-alloc_size-attribute-pg_trgm.patchDownload+11-3
0004-Fixes-for-alloc_size-attribute-numeric.patchtext/plain; charset=UTF-8; name=0004-Fixes-for-alloc_size-attribute-numeric.patchDownload+5-2
0005-XXX-ATTRIBUTE_FIXED_PART_SIZE-fix.patchtext/plain; charset=UTF-8; name=0005-XXX-ATTRIBUTE_FIXED_PART_SIZE-fix.patchDownload+1-2
Hello
#define SET_VARSIZE_4B_C(PTR,len) \
- (((varattrib_4b *) (PTR))->va_4byte.va_header = ((len) & 0x3FFFFFFF) | 0x40000000)
+ (((varattrib_4b_normal *) (PTR))->va_4byte.va_header = ((len) & 0x3FFFFFFF) | 0x40000000)
That seems to be a missed removal? Should be ->va_header
On Fri Nov 13, 2026 at 11:51 AM UTC, Peter Eisentraut wrote:
In my ventures to apply more compiler attributes, I came to alloc_size.
This is to annotate memory allocation functions so that the compiler can
know the size of the allocated memory and so some cross-checking based
on that. This is all pretty straightforward and I think we should do it.Andres had previously reported some issues he encountered in [0]. (That
message didn't go into details on what attributes or compiler options
were used, but it certainly looked very closely related.)The issue is that under-allocating a struct is bad. With this
attribute, the compiler flags this, because it looks like a mistake.
For uses where this is legitimate, you'd normally use a struct with a
trailing flexible-array member, and we mostly make use of that. But
there are a couple of cases where we assign to a struct pointer less
memory than what the struct expects. In some cases, this is done with a
union where we only allocate enough memory for one branch of the union,
which seems like an intuitive thing to do but is actually also bad.In the attached patches I have some fixes and/or workarounds for the
issues that I found.Note that as documented this attribute and the associated warning
options are for diagnostics only, they don't change compilation. So in
that sense, once all warnings are fixed, this shouldn't create any more
hidden issues.- varattrib_4b is a union of a normal varlena with a 4-byte header and 0
or more payload bytes, and a compressed format with a 4-byte header, a
4-byte "tcinfo", followed by payload data. The typical coding patternval = palloc(VARHDRSZ + len);
SET_VARSIZE(val, VARHDRSZ + len);could get flagged by -Warray-bounds because the allocation could be too
short to cover both branches of the union. In this case, I think we
don't actually need the union, so I'm replacing this with two separate
structs.- TRGM in contrib/pg_trgm/ is an instance of what [0] talks about: It
has a trailing flexible array member, but it also has trailing padding,
so allocating palloc(TRGMHDRSIZE) causes a warning, because it's less
than sizeof(TRGM). In my patch I work around this by allocating like
palloc(Max(TRGMHDRSIZE, sizeof(TRGM))).
Maybe we should add an allocator function or macro for this? In the
counted_by discussion, I talked a bit about how core does a pretty good
job of providing allocator helpers for structs with flexible array
members. I realize this is contrib/, but it might still be valuable.
- There is a bit of a maze of different structs and unions that underlie
the numeric type. The allocation (Numeric) palloc(NUMERIC_HDRSZ_SHORT)
gets flagged by the compiler. In my patch I change this to (Numeric)
palloc(sizeof(struct NumericData)), which allocates a bit more but makes
the sizes and types match. Maybe some more judicious reshuffling of the
underlying data structures could avoid this, or maybe it's okay like this.
Like the previous comment, I wonder if some helper allocator
functions/macros could be useful. But as you said, maybe some
reshuffling could avoid that all together, not sure.
- As of commit 6a3da13d376 from this week, FormData_pg_attribute has 4
bytes of trailing padding, and so sizeof(FormData_pg_attribute) is now
different from ATTRIBUTE_FIXED_PART_SIZE. This causes a warning in one
place. We had a discussion about this some time ago [1], where the
opinions diverged between like, it's important to keep this distinction
in case we ever have padding bytes, and, it's already incorrect and
would be even more incorrect if we had padding bytes. ;-) Well, we're
here now, so perhaps this should be revisited.
In [1], Tom said:
What I'm trying to figure out is if we need to put in place some
provisions to prevent there from being pad bytes at the end of any
catalog struct.
I figured I could propose at least one way to flag padding bytes in the
attached patch. I am currently not proposing a solution.
Two options proposed in the thread[1]:
- Placeholder fields exactly matching the size of the padding bytes.
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree ev_qual;
pg_node_tree ev_action;
#else
uint8 dummy;
#endif
- Use different structs for on-media and in-memory representations.
To not rock the boat too much, maybe I would throw my hat in the ring
for option 1? It seems like the simplest and most immediate solution.
This is all that I found. [0] lists a lot more sources of potential
problems, but apparently they are not flagged as they are currently used.[0]:
/messages/by-id/axxbw7ku4w24hwe6uvkm4vdufo4zr4k5344m6naahkpsoafag5@llovwgptzbgq[1]:
/messages/by-id/7c7d41c1-fd5d-f9a5-afd5-f00168d91fe2@2ndquadrant.com
Anyway, thanks for the patch. I think the attribute is valuable, and
it's good that we are having this discussion.
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
Attachments:
nocfbot-v1-trailing-padding-test.patchtext/x-patch; charset=utf-8; name=nocfbot-v1-trailing-padding-test.patchDownload+241-2
On 13.11.26 15:39, Zsolt Parragi wrote:
#define SET_VARSIZE_4B_C(PTR,len) \ - (((varattrib_4b *) (PTR))->va_4byte.va_header = ((len) & 0x3FFFFFFF) | 0x40000000) + (((varattrib_4b_normal *) (PTR))->va_4byte.va_header = ((len) & 0x3FFFFFFF) | 0x40000000)That seems to be a missed removal? Should be ->va_header
Yes, this is in the big-endian branch that I did not test. I will fix
this in the next patch set. Thanks.
On 16.11.26 18:49, Tristan Partin wrote:
- TRGM in contrib/pg_trgm/ is an instance of what [0] talks about: It
has a trailing flexible array member, but it also has trailing padding,
so allocating palloc(TRGMHDRSIZE) causes a warning, because it's less
than sizeof(TRGM). In my patch I work around this by allocating like
palloc(Max(TRGMHDRSIZE, sizeof(TRGM))).Maybe we should add an allocator function or macro for this? In the
counted_by discussion, I talked a bit about how core does a pretty good
job of providing allocator helpers for structs with flexible array
members. I realize this is contrib/, but it might still be valuable.- There is a bit of a maze of different structs and unions that underlie
the numeric type. The allocation (Numeric) palloc(NUMERIC_HDRSZ_SHORT)
gets flagged by the compiler. In my patch I change this to (Numeric)
palloc(sizeof(struct NumericData)), which allocates a bit more but makes
the sizes and types match. Maybe some more judicious reshuffling of the
underlying data structures could avoid this, or maybe it's okay like this.Like the previous comment, I wonder if some helper allocator
functions/macros could be useful. But as you said, maybe some
reshuffling could avoid that all together, not sure.
In the attached patch series, I have an idea for a more generalized
approach: A macro that computes the required size and then a wrapper
macro around it for allocating. This then can be applied to any
structure with a flexible array member, and it eliminates some
duplication with the offsetof calls and gets you the kind of type safety
that palloc_object and palloc_array provide.
I have also added a patch that makes some more uses of this macro.
These are the same kinds of problems as the one I showed for pg_trgm and
numeric, except that the size isn't constant and so the compiler won't
flag it.
- As of commit 6a3da13d376 from this week, FormData_pg_attribute has 4
bytes of trailing padding, and so sizeof(FormData_pg_attribute) is now
different from ATTRIBUTE_FIXED_PART_SIZE. This causes a warning in one
place. We had a discussion about this some time ago [1], where the
opinions diverged between like, it's important to keep this distinction
in case we ever have padding bytes, and, it's already incorrect and
would be even more incorrect if we had padding bytes. ;-) Well, we're
here now, so perhaps this should be revisited.
I have adjusted the description of this patch a bit. There are a lot of
details to be figured out in this area, but at least we can fix this
particular issue easily and in a consistent way. tupdesc.c already uses
sizeof(FormData_pg_attribute) consistently for allocating pg_attribute
structures. It only uses ATTRIBUTE_FIXED_PART_SIZE for things like
memcpy(). With the proposed patch, we make bootstrap.c consistent in
this regard, and we make the warning go away. The deeper issues can
then be researched independent of this.
In [1], Tom said:
What I'm trying to figure out is if we need to put in place some
provisions to prevent there from being pad bytes at the end of any
catalog struct.I figured I could propose at least one way to flag padding bytes in the
attached patch. I am currently not proposing a solution.Two options proposed in the thread[1]:
- Placeholder fields exactly matching the size of the padding bytes.
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree ev_qual;
pg_node_tree ev_action;
#else
uint8 dummy;
#endif- Use different structs for on-media and in-memory representations.
To not rock the boat too much, maybe I would throw my hat in the ring
for option 1? It seems like the simplest and most immediate solution.
I think we could get the same check simpler and more portably by having
genbki.pl generate some static assertions that check like
static_assert(sizeof(FormData_pg_attribute) ==
offsetof(FormData_pg_attribute, attcollation) +
sizeof(FormData_pg_attribute.attcollation));
But this flags a bunch of issues. Sorting that out might be a big project.
Attachments:
t259206_5v2-0001-Apply-alloc_size-attribute-to-memory-allocation-f.patchtext/plain; charset=UTF-8; name=v2-0001-Apply-alloc_size-attribute-to-memory-allocation-f.patchDownload+56-40
v2-0002-Fixes-for-alloc_size-attribute-varatt.patchtext/plain; charset=UTF-8; name=v2-0002-Fixes-for-alloc_size-attribute-varatt.patchDownload+34-29
v2-0003-Fixes-for-alloc_size-attribute-pg_trgm.patchtext/plain; charset=UTF-8; name=v2-0003-Fixes-for-alloc_size-attribute-pg_trgm.patchDownload+11-3
v2-0004-Fixes-for-alloc_size-attribute-numeric.patchtext/plain; charset=UTF-8; name=v2-0004-Fixes-for-alloc_size-attribute-numeric.patchDownload+5-2
v2-0005-Fixes-for-alloc_size-attribute-bootstrap.patchtext/plain; charset=UTF-8; name=v2-0005-Fixes-for-alloc_size-attribute-bootstrap.patchDownload+3-4
v2-0006-WIP-flex_size.patchtext/plain; charset=UTF-8; name=v2-0006-WIP-flex_size.patchDownload+39-13
v2-0007-WIP-More-flex_size-candidates.patchtext/plain; charset=UTF-8; name=v2-0007-WIP-More-flex_size-candidates.patchDownload+13-20
On Fri Nov 20, 2026 at 3:34 PM UTC, Peter Eisentraut wrote:
On 16.11.26 18:49, Tristan Partin wrote:
- TRGM in contrib/pg_trgm/ is an instance of what [0] talks about: It
has a trailing flexible array member, but it also has trailing padding,
so allocating palloc(TRGMHDRSIZE) causes a warning, because it's less
than sizeof(TRGM). In my patch I work around this by allocating like
palloc(Max(TRGMHDRSIZE, sizeof(TRGM))).Maybe we should add an allocator function or macro for this? In the
counted_by discussion, I talked a bit about how core does a pretty good
job of providing allocator helpers for structs with flexible array
members. I realize this is contrib/, but it might still be valuable.- There is a bit of a maze of different structs and unions that underlie
the numeric type. The allocation (Numeric) palloc(NUMERIC_HDRSZ_SHORT)
gets flagged by the compiler. In my patch I change this to (Numeric)
palloc(sizeof(struct NumericData)), which allocates a bit more but makes
the sizes and types match. Maybe some more judicious reshuffling of the
underlying data structures could avoid this, or maybe it's okay like this.Like the previous comment, I wonder if some helper allocator
functions/macros could be useful. But as you said, maybe some
reshuffling could avoid that all together, not sure.In the attached patch series, I have an idea for a more generalized
approach: A macro that computes the required size and then a wrapper
macro around it for allocating. This then can be applied to any
structure with a flexible array member, and it eliminates some
duplication with the offsetof calls and gets you the kind of type safety
that palloc_object and palloc_array provide.I have also added a patch that makes some more uses of this macro.
These are the same kinds of problems as the one I showed for pg_trgm and
numeric, except that the size isn't constant and so the compiler won't
flag it.
Overall, I think the concept looks good! Maybe adding
a palloc_empty_flex_array() macro could be useful to make the
0 allocations a little bit nicer to read.
@@ -1226,6 +1226,24 @@ pg_noreturn extern void ExceptionalCondition(const char *conditionName,
*/
#define Min(x, y) ((x) < (y) ? (x) : (y))+/* + * flex_size + * Return the size of a struct "type" whose flexible array member + * "member" holds "count" elements. + * + * The result is never less than sizeof(type), so that any trailing padding + * of the struct is covered. "count" is evaluated only once. + */ +#define flex_size(type, member, count) \ + flex_size_impl(sizeof(type), offsetof(type, member), \ + sizeof(((type *) 0)->member[0]), (count)) + +static inline size_t +flex_size_impl(size_t structsz, size_t offset, size_t elemsz, size_t count) +{ + return Max(structsz, offset + count * elemsz); +} +/* Get a bit mask of the bits set in non-size_t aligned addresses */
#define SIZE_T_ALIGN_MASK (sizeof(size_t) - 1)
It might behoove us to use the same safe math techniques that we use in
palloc_mul() here. Maybe I am being too cautious. It is not clear to me
if any potential "count" values would ever be derived from user input.
It would protect from us _succeeding_ to allocate a potentially smaller
amount of memory, and then writing or reading past the end.
- As of commit 6a3da13d376 from this week, FormData_pg_attribute has 4
bytes of trailing padding, and so sizeof(FormData_pg_attribute) is now
different from ATTRIBUTE_FIXED_PART_SIZE. This causes a warning in one
place. We had a discussion about this some time ago [1], where the
opinions diverged between like, it's important to keep this distinction
in case we ever have padding bytes, and, it's already incorrect and
would be even more incorrect if we had padding bytes. ;-) Well, we're
here now, so perhaps this should be revisited.I have adjusted the description of this patch a bit. There are a lot of
details to be figured out in this area, but at least we can fix this
particular issue easily and in a consistent way. tupdesc.c already uses
sizeof(FormData_pg_attribute) consistently for allocating pg_attribute
structures. It only uses ATTRIBUTE_FIXED_PART_SIZE for things like
memcpy(). With the proposed patch, we make bootstrap.c consistent in
this regard, and we make the warning go away. The deeper issues can
then be researched independent of this.
I agree that this problem is likely orthogonal, and we can move it to
a separate discussion.
In [1], Tom said:
What I'm trying to figure out is if we need to put in place some
provisions to prevent there from being pad bytes at the end of any
catalog struct.I figured I could propose at least one way to flag padding bytes in the
attached patch. I am currently not proposing a solution.Two options proposed in the thread[1]:
- Placeholder fields exactly matching the size of the padding bytes.
#ifdef CATALOG_VARLEN /* variable-length fields start here */
pg_node_tree ev_qual;
pg_node_tree ev_action;
#else
uint8 dummy;
#endif- Use different structs for on-media and in-memory representations.
To not rock the boat too much, maybe I would throw my hat in the ring
for option 1? It seems like the simplest and most immediate solution.I think we could get the same check simpler and more portably by having
genbki.pl generate some static assertions that check likestatic_assert(sizeof(FormData_pg_attribute) ==
offsetof(FormData_pg_attribute, attcollation) +
sizeof(FormData_pg_attribute.attcollation));But this flags a bunch of issues. Sorting that out might be a big project.
I had a similar idea as well.
Overall, I think that the non-WIP patches look good, but it would
probably be useful to get another committer's thoughts before moving
forward.
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)