pgsql: Optimize tuple deformation
Optimize tuple deformation
This commit includes various optimizations to improve the performance of
tuple deformation.
We now precalculate CompactAttribute's attcacheoff, which allows us to
remove the code from the deform routines which was setting the
attcacheoff. Setting the attcacheoff is now handled by
TupleDescFinalize(), which must be called before the TupleDesc is used for
anything. Having TupleDescFinalize() means we can store the first
attribute in the TupleDesc which does not have an offset cached. That
allows us to add a dedicated deforming loop to deform all attributes up
to the final one with an attcacheoff set, or up to the first NULL
attribute, whichever comes first.
Here we also improve tuple deformation performance of tuples with NULLs.
Previously, if the HEAP_HASNULL bit was set in the tuple's t_infomask,
deforming would, one-by-one, check each and every bit in the NULL bitmap
to see if it was zero. Now, we process the NULL bitmap 1 byte at a time
rather than 1 bit at a time to find the attnum with the first NULL. We
can now deform the tuple without checking for NULLs up to just before that
attribute.
We also record the maximum attribute number which is guaranteed to exist
in the tuple, that is, has a NOT NULL constraint and isn't an
atthasmissing attribute. When deforming only attributes prior to the
guaranteed attnum, we've no need to access the tuple's natt count. As an
additional optimization, we only count fixed-width columns when
calculating the maximum guaranteed column, as this eliminates the need to
emit code to fetch byref types in the deformation loop for guaranteed
attributes.
Some locations in the code deform tuples that have yet to go through NOT
NULL constraint validation. We're unable to perform the guaranteed
attribute optimization when that's the case. This optimization is opt-in
via the TupleTableSlot using the TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS
flag.
This commit also adds a more efficient way of populating the isnull
array by using a bit-wise SWAR trick which performs multiplication on the
inverse of the tuple's bitmap byte and masking out all but the lower bit
of each of the boolean's byte. This results in much more optimal code
when compared to determining the NULLness via att_isnull(). 8 isnull
elements are processed at once using this method, which means we need to
round the tts_isnull array size up to the next 8 bytes. The palloc code
does this anyway, but the round-up needed to be formalized so as not to
overwrite the sentinel byte in MEMORY_CONTEXT_CHECKING builds. Doing
this also allows the NULL-checking deforming loop to more efficiently
check the isnull array, rather than doing the bit-wise processing for each
attribute that att_isnull() does.
The level of performance improvement from these changes seems to vary
depending on the CPU architecture. Apple's M chips seem particularly
fond of the changes, with some of the tested deform-heavy queries going
over twice as fast as before. With x86-64, the speedups aren't quite as
large. With tables containing only a small number of columns, the
speedups will be less.
Author: David Rowley <dgrowleyml@gmail.com>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: John Naylor <johncnaylorls@gmail.com>
Reviewed-by: Amit Langote <amitlangote09@gmail.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Reviewed-by: Álvaro Herrera <alvherre@kurilemu.de>
Reviewed-by: Junwang Zhao <zhjwpku@gmail.com>
Discussion: /messages/by-id/CAApHDvpoFjaj3+w_jD5uPnGazaw41A71tVJokLDJg2zfcigpMQ@mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/c456e39113809376f6604e720910ccd24e18e034
Modified Files
--------------
src/backend/access/common/heaptuple.c | 360 +++++++++------------
src/backend/access/common/indextuple.c | 363 +++++++++------------
src/backend/access/common/tupdesc.c | 51 +++
src/backend/access/spgist/spgutils.c | 3 -
src/backend/executor/execMain.c | 8 +-
src/backend/executor/execTuples.c | 430 ++++++++++++++-----------
src/backend/executor/execUtils.c | 2 +-
src/backend/executor/nodeAgg.c | 2 +-
src/backend/executor/nodeBitmapHeapscan.c | 3 +-
src/backend/executor/nodeCtescan.c | 2 +-
src/backend/executor/nodeCustom.c | 4 +-
src/backend/executor/nodeForeignscan.c | 4 +-
src/backend/executor/nodeFunctionscan.c | 2 +-
src/backend/executor/nodeIndexonlyscan.c | 5 +-
src/backend/executor/nodeIndexscan.c | 3 +-
src/backend/executor/nodeNamedtuplestorescan.c | 2 +-
src/backend/executor/nodeSamplescan.c | 3 +-
src/backend/executor/nodeSeqscan.c | 3 +-
src/backend/executor/nodeSubqueryscan.c | 3 +-
src/backend/executor/nodeTableFuncscan.c | 2 +-
src/backend/executor/nodeTidrangescan.c | 3 +-
src/backend/executor/nodeTidscan.c | 3 +-
src/backend/executor/nodeValuesscan.c | 2 +-
src/backend/executor/nodeWorktablescan.c | 2 +-
src/backend/jit/llvm/llvmjit_deform.c | 6 -
src/backend/replication/pgoutput/pgoutput.c | 4 +-
src/backend/utils/cache/relcache.c | 12 -
src/include/access/tupdesc.h | 20 +-
src/include/access/tupmacs.h | 224 ++++++++++++-
src/include/executor/executor.h | 3 +-
src/include/executor/tuptable.h | 34 +-
31 files changed, 902 insertions(+), 666 deletions(-)
On 15.03.26 23:50, David Rowley wrote:
Optimize tuple deformation
This commit c456e391138 changed in slot_deform_heap_tuple() the type of
attnum from int to size_t, which seems unusual and doesn't fit with the
surrounding code. Could you check whether that was intentional?
On Wed, 7 Oct 2026 at 19:31, Peter Eisentraut <peter@eisentraut.org> wrote:
On 15.03.26 23:50, David Rowley wrote:
Optimize tuple deformation
This commit c456e391138 changed in slot_deform_heap_tuple() the type of
attnum from int to size_t, which seems unusual and doesn't fit with the
surrounding code. Could you check whether that was intentional?
It was done to work around a problem with gcc where it was refusing to
index the CompactAttribute array by adding 8 to the previous address
to get the address of the next element. Instead, it was doing LEA on
each loop, which was slower due to shift and add rather than just add.
The best I could figure out is that gcc didn't want to optimise to use
LEA due to concerns about the integer wrapping around, which, if it
did, would be a different address than doing 2^31 adds on a 64-bit
pointer.
David
On 07.10.26 10:54, David Rowley wrote:
On Wed, 7 Oct 2026 at 19:31, Peter Eisentraut <peter@eisentraut.org> wrote:
On 15.03.26 23:50, David Rowley wrote:
Optimize tuple deformation
This commit c456e391138 changed in slot_deform_heap_tuple() the type of
attnum from int to size_t, which seems unusual and doesn't fit with the
surrounding code. Could you check whether that was intentional?It was done to work around a problem with gcc where it was refusing to
index the CompactAttribute array by adding 8 to the previous address
to get the address of the next element. Instead, it was doing LEA on
each loop, which was slower due to shift and add rather than just add.
The best I could figure out is that gcc didn't want to optimise to use
LEA due to concerns about the integer wrapping around, which, if it
did, would be a different address than doing 2^31 adds on a 64-bit
pointer.
Oh! :-/
How about a really big comment about that, and maybe change the type
from size_t to uint64 (or maybe uintptr_t, if it's supposed to vary on
32-bit platforms), so it looks more intentional and not like a mistake?
On Fri, 9 Oct 2026 at 04:44, Peter Eisentraut <peter@eisentraut.org> wrote:
On 07.10.26 10:54, David Rowley wrote:
It was done to work around a problem with gcc where it was refusing to
index the CompactAttribute array by adding 8 to the previous address
to get the address of the next element. Instead, it was doing LEA on
each loop, which was slower due to shift and add rather than just add.
The best I could figure out is that gcc didn't want to optimise to use
LEA due to concerns about the integer wrapping around, which, if it
did, would be a different address than doing 2^31 adds on a 64-bit
pointer.Oh! :-/
How about a really big comment about that, and maybe change the type
from size_t to uint64 (or maybe uintptr_t, if it's supposed to vary on
32-bit platforms), so it looks more intentional and not like a mistake?
I don't object to having more details there. The problem was that I
just wasn't that certain why gcc was doing this. The above is just a
theory which might be incorrect. When I mock up trivial examples in
godbolt.org, I can't recreate the problem, so it may have something to
do with register pressure because there are so many things in flight
that there's a shortage of registers to store required values.
I'd like to get time to put in more effort into figuring out what's
going on with the gcc as I wondered how many other places we're
getting inefficient code because of this. More research may discover
that we really shouldn't use ints to index arrays, for example.
It is meant to be 32-bits on 32-bit platforms. Why is uintptr_t better
than size_t for indexing an array? Or rather, what's wrong with
size_t? I had assumed you highlighted this because of some fields
being int and some being size_t, but if you're happier with uintptr_t,
then I may have misunderstood.
David
On 08.10.26 22:35, David Rowley wrote:
I'd like to get time to put in more effort into figuring out what's
going on with the gcc as I wondered how many other places we're
getting inefficient code because of this. More research may discover
that we really shouldn't use ints to index arrays, for example.It is meant to be 32-bits on 32-bit platforms. Why is uintptr_t better
than size_t for indexing an array? Or rather, what's wrong with
size_t? I had assumed you highlighted this because of some fields
being int and some being size_t, but if you're happier with uintptr_t,
then I may have misunderstood.
I'm fine with leaving it as is for now. I'm going around fixing signed
vs. unsigned integer mismatches, and this is being flagged because
size_t is unsigned and attribute numbers are in general signed, and
could at least in some cases actually be negative. But this is only one
of many problems, so I can silence this in my local work for now. (So
the suggestion to use uintptr_t was also wrong; it should have been
intptr_t, or possibly ssize_t, but given your explanation, I'm not
worried about this right now; I would rather see you find a proper
solution.)