Make memory checking / sanitizing infrastructure better
Hi,
Currently, around $subject, we have a few weaknesses:
1) For small-ish power-of-two allocations, aset.c does not put a sentinel at
the end of the allocation. Given how common power-of-two allocations are,
that seems decidedly suboptimal.
2) The sentinel we use is just one byte. This means that basic variable
alignment rules will often lead to access-beyond-end-of-allocation to be
further out, preventing us from detecting that.
3) If we do detect memory context issues, we just issue WARNINGs, we do not
crash. I think there are a few different problems with that approach:
often there will eventually be a crashes, due to the memory corruption that
already occurred, but it'll be further away from where we detected the
problem, rather than failing closer to the issue.
Another big issue is that this means that in contexts where we do not check
for WARNINGs, e.g. tap tests or in background processes, such WARNINGs
won't trigger test failures. Which seems rather bad to me.
A third issue is that the warnings are useless for detecting problems
during automated testing like fuzzing. It's very normal for some WARNINGs
to be issued when e.g. datatype input functions, something like memory
corruption really needs to result in a crash to be properly fuzzable.
4) We support using address sanitizer, but due to address sanitizer only
learning about AllocBlock (et al) level allocations, many read accesses
beyond the end of an allocation will never be detected.
5) Normally address sanitizers quarantines individual allocation "addresses"
for a while, to make it more likely to catch use-after-free issues.
Without something like that, many types of issues won't be detected.
I think these aren't actually *that* hard to solve:
1) The reason we don't always use the sentinel for power-of-two allocations is
that naively doing so would interfere with the size classing for the
freelist. But that's relatively easy to address - we can "just" add the
space for the size classes *after* determining the size class. IIRC we
used to have a similar issue with the per-allocation chunk header and
solved it this way too.
2) We should make make the sentinel size configurable and default to either 8
or 16 bytes.
3) I think we should make the memory context failures crash. Perhaps by
emitting WARNINGs and then crashing if any of them occurred.
As we trigger memory context checking during commit/abort handling, just
using PANIC won't always reliably work, E.g. the AllocSetCheck() in
AllocSetReset(), can trigger recursive PANICs due to the
MemoryContextReset() in errstart(), if the corruption in in ErrorContext.
Whether that's a real issue worth worrying about, IDK.
4) For this I prototyped making the valgrind annotations more generic and
using the address sanitizer interface to mark memory as poisoned /
unpoisoned. That doesn't provide quite all the checking that valgrind can
do (it doesn't track uninitialized memory), but it's considerably better
than our memory context checking, and *much* *much* faster than valgrind.
5) I think we should add a mode to all our allocators that just turns every
allocation into a large allocation (perhaps with something slightly
different for slab). In that mode asan, valgrind, et al would be more
precise. Of course that'll have rather deleterious performance effects,
but I think it'd still be quite useful for debugging.
I have prototype patches for 1-4, but wanted to hear about whether such
changes sound sane to others, before spending time to polish all of this up.
Greetings,
Andres Freund
On Thu, May 28, 2026 at 9:07 AM Andres Freund <andres@anarazel.de> wrote:
4) For this I prototyped making the valgrind annotations more generic and
using the address sanitizer interface to mark memory as poisoned /
unpoisoned. That doesn't provide quite all the checking that valgrind can
do (it doesn't track uninitialized memory), but it's considerably better
than our memory context checking, and *much* *much* faster than valgrind.
Neat! I was recently hoping for this in [1]/messages/by-id/CAOYmi+nCg5upBVOo_UCSjMfO=YMkZXcSEsgaADKXqerr5wahZQ@mail.gmail.com.
I have prototype patches for 1-4, but wanted to hear about whether such
changes sound sane to others, before spending time to polish all of this up.
They all sound like nice improvements to me. I like the idea in (5), too.
--Jacob
[1]: /messages/by-id/CAOYmi+nCg5upBVOo_UCSjMfO=YMkZXcSEsgaADKXqerr5wahZQ@mail.gmail.com
On 28.05.26 18:07, Andres Freund wrote:
I have prototype patches for 1-4, but wanted to hear about whether such
changes sound sane to others, before spending time to polish all of this up.
This all sounds entirely reasonable.
On Thu, May 28, 2026 at 9:37 PM Andres Freund <andres@anarazel.de> wrote:
I think these aren't actually *that* hard to solve:
1) The reason we don't always use the sentinel for power-of-two allocations is
that naively doing so would interfere with the size classing for the
freelist. But that's relatively easy to address - we can "just" add the
space for the size classes *after* determining the size class. IIRC we
used to have a similar issue with the per-allocation chunk header and
solved it this way too.2) We should make make the sentinel size configurable and default to either 8
or 16 bytes.
+1
3) I think we should make the memory context failures crash. Perhaps by
emitting WARNINGs and then crashing if any of them occurred.As we trigger memory context checking during commit/abort handling, just
using PANIC won't always reliably work, E.g. the AllocSetCheck() in
AllocSetReset(), can trigger recursive PANICs due to the
MemoryContextReset() in errstart(), if the corruption in in ErrorContext.
Whether that's a real issue worth worrying about, IDK.4) For this I prototyped making the valgrind annotations more generic and
using the address sanitizer interface to mark memory as poisoned /
unpoisoned. That doesn't provide quite all the checking that valgrind can
do (it doesn't track uninitialized memory), but it's considerably better
than our memory context checking, and *much* *much* faster than valgrind.
IIUC, you are proposing to use sanitizer interface to implement
crashing behaviour mentioned in 3? Or these two are separate things?
To me both of them seem to achieve the same end result.
5) I think we should add a mode to all our allocators that just turns every
allocation into a large allocation (perhaps with something slightly
different for slab). In that mode asan, valgrind, et al would be more
precise. Of course that'll have rather deleterious performance effects,
but I think it'd still be quite useful for debugging.
I think this has an advantage of its own, we will benefit from memory
sanity check improvements in system calls without necessarily
implementing them in PostgreSQL and also use-after-free failure not
necessarily covered by other proposals.
--
Best Wishes,
Ashutosh Bapat
Hi,
On 2026-10-08 14:56:24 +0530, Ashutosh Bapat wrote:
On Thu, May 28, 2026 at 9:37 PM Andres Freund <andres@anarazel.de> wrote:
3) I think we should make the memory context failures crash. Perhaps by
emitting WARNINGs and then crashing if any of them occurred.As we trigger memory context checking during commit/abort handling, just
using PANIC won't always reliably work, E.g. the AllocSetCheck() in
AllocSetReset(), can trigger recursive PANICs due to the
MemoryContextReset() in errstart(), if the corruption in in ErrorContext.
Whether that's a real issue worth worrying about, IDK.4) For this I prototyped making the valgrind annotations more generic and
using the address sanitizer interface to mark memory as poisoned /
unpoisoned. That doesn't provide quite all the checking that valgrind can
do (it doesn't track uninitialized memory), but it's considerably better
than our memory context checking, and *much* *much* faster than valgrind.IIUC, you are proposing to use sanitizer interface to implement
crashing behaviour mentioned in 3? Or these two are separate things?
Separate. Using asan is pretty expensive (~2-3x slower tests, considerably
higher memory usage), there's absolutely no reason to not have our own memory
context checking infrastructure only really be usable if you use sanitizers.
I think there's also corruption that only our internal checking will find.
Running all tests without asan:
real 1m34.240s
user 6m27.832s
sys 4m20.235s
with asan:
real 3m31.336s
user 12m44.840s
sys 9m13.033s
5) I think we should add a mode to all our allocators that just turns every
allocation into a large allocation (perhaps with something slightly
different for slab). In that mode asan, valgrind, et al would be more
precise. Of course that'll have rather deleterious performance effects,
but I think it'd still be quite useful for debugging.I think this has an advantage of its own, we will benefit from memory
sanity check improvements in system calls without necessarily
implementing them in PostgreSQL and also use-after-free failure not
necessarily covered by other proposals.
I'm not really following, which memory sanity checking in system calls are you
thinking of?
FWIW, I'd started hacking on the asan piece a while back, here's my last WIP
patches on this topic. I ran out of time to work on this at the time, and
haven't found my way back since. No guarantees that anything therein is
correct.
Greetings,
Andres Freund
Attachments:
t139776_5v2-0001-WIP-mmgr-Use-larger-and-guaranteed-to-exist-senti.patchtext/x-diff; charset=us-asciiDownload+88-89
v2-0002-WIP-mmgr-Improve-asan-support-for-individual-allo.patchtext/x-diff; charset=us-asciiDownload+345-272
v2-0003-Fix-spurious-valgrind-errors-within-NUMA-views.patchtext/x-diff; charset=us-asciiDownload+10-1
On Fri, Oct 9, 2026 at 9:37 PM Andres Freund <andres@anarazel.de> wrote:
5) I think we should add a mode to all our allocators that just turns every
allocation into a large allocation (perhaps with something slightly
different for slab). In that mode asan, valgrind, et al would be more
precise. Of course that'll have rather deleterious performance effects,
but I think it'd still be quite useful for debugging.I think this has an advantage of its own, we will benefit from memory
sanity check improvements in system calls without necessarily
implementing them in PostgreSQL and also use-after-free failure not
necessarily covered by other proposals.I'm not really following, which memory sanity checking in system calls are you
thinking of?
I used “system calls” incorrectly; I meant the underlying allocator
and the memory-debugging infrastructure around its allocation
interfaces.
The point I was trying to make was slightly broader than the immediate
increase in ASAN/Valgrind precision. PostgreSQL’s memory-context
infrastructure and MEMORY_CONTEXT_CHECKING date from 2000. Valgrind
appeared later, ASAN much later, and allocator/tooling features such
as redzones, quarantine and other hardening have continued to improve
since then.
If, in this mode, each palloc() is represented by a separate
underlying allocation, its boundary is visible directly to that
allocator and to tools interposing on malloc()/free(). PostgreSQL
could therefore benefit "immediately" from future improvements in
those allocators and tools without necessarily having to reproduce
each new check in every memory-context implementation. If we had this
mode today, we would automatically benefit from use-after-free
checking.
Of course, this only applies to checks that operate at the underlying
allocation boundary, and your proposed ASAN annotations would cover
many of the same cases. So this is probably not an advantage separate
from the “ASAN, Valgrind, et al would be more precise” point you
already made. It is more the future-proofing aspect of exposing each
allocation to that infrastructure. Is that already what you had in
mind?
Thanks for sharing patches, will take a look at them soon.
--
Best Wishes,
Ashutosh Bapat