glist: _Generic wrapper for selective dlist/dclist usage
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253644psql -h localhost -U postgresBuilt from patchset v1 (message #1), September 21, 2026 at 04:52 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253644_1 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253644_1 && git checkout t253644_1Patchset v1 (message #1) is on t253644_1
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .
Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1]/messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com.
I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
[^0]: This case for builds with MEMORY_CONTEXT_CHECKING, but builds
with USE_ASSERT_CHECKING or WRITE_READ_PARSE_PLAN_TREES -like options
may also want this.
[1]: /messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com
On Wed, 02 Sep 2026 at 13:36, Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.
+1 for this idea.
Should we adopt the new glist interface in the existing code to better
showcase its intended use?
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)[^0]: This case for builds with MEMORY_CONTEXT_CHECKING, but builds
with USE_ASSERT_CHECKING or WRITE_READ_PARSE_PLAN_TREES -like options
may also want this.[1]: /messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com
--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.
On Wed, 2 Sept 2026 at 17:32, Japin Li <japinli@hotmail.com> wrote:
On Wed, 02 Sep 2026 at 13:36, Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.+1 for this idea.
Should we adopt the new glist interface in the existing code to better
showcase its intended use?
Did you have a place in mind where we have dlist and dclist in use to
track the same information in different build configurations? I don't
like churning code for the sake of new APIs, especially when those new
APIs are not strictly better than the current ones.
Note that I add glist to solve ergonomics issues when you want to
handle both types of lists. It's not meant to be used as default API
to fields that are always of a dlist type, or always of a dclist type,
even if it could be used like that.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
On Wed Sep 2, 2026 at 11:36 AM UTC, Matthias van de Meent wrote:
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)[^0]: This case for builds with MEMORY_CONTEXT_CHECKING, but builds
with USE_ASSERT_CHECKING or WRITE_READ_PARSE_PLAN_TREES -like options
may also want this.[1]: /messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com
I can't speak much to the purpose, but the patch itself looks correct
given my understanding of _Generic. I also checked that the dlist/dclist
function names match up with the glist function names. Do you think it
makes sense to add a comment of how one might use glist? There is
a large comment at the top of ilist.h where it could make sense to add
one if you think it would be useful. Maybe dglist would be a better name
to keep it more scoped to doubly linked lists?
--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)
On Wed, 2 Sept 2026 at 23:36, Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:
Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].
This feels like putting the cart before the horse. What purpose does
maintaining a count in the dlist serve at all for [1]? Last I looked
at the 0004 patch on [1] you were only counting and asserting the
count in debug builds. Since the count is not used for anything else,
it seemed to me that all the assert was doing was verifying the count
tracking was correct, and since that count tracking code only existed
in debug builds, there was no point in it.
I think if you have a concrete proposal for your 0001 patch here,
there should be a 0002 patch which uses the new macros for some
legitimate reason. Otherwise, if the only legit reason is in [1], then
this patch should be part of that series. At the moment, all this
thread proposes to do is introduce dead code.
[1]: /messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com
David
On Wed, 2 Sept 2026 at 19:35, Tristan Partin <tristan@partin.io> wrote:
On Wed Sep 2, 2026 at 11:36 AM UTC, Matthias van de Meent wrote:
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)[^0]: This case for builds with MEMORY_CONTEXT_CHECKING, but builds
with USE_ASSERT_CHECKING or WRITE_READ_PARSE_PLAN_TREES -like options
may also want this.[1]: /messages/by-id/CAEze2WiPyruOtUOSyRUV8mQssjmYwno0M6hkxC_iUpH-=W8WcA@mail.gmail.com
I can't speak much to the purpose, but the patch itself looks correct
given my understanding of _Generic. I also checked that the dlist/dclist
function names match up with the glist function names. Do you think it
makes sense to add a comment of how one might use glist? There is
a large comment at the top of ilist.h where it could make sense to add
one if you think it would be useful.
Good idea.
Maybe dglist would be a better name
to keep it more scoped to doubly linked lists?
I'd like to avoid painting ourselves into a corner here; though I have
no current need for slist support, I also don't want to rule out that
someone else can find a need for it, and I think adding another copy
of the macros (instead of adapting existing ones) would be a shame.
So here I'd like to keep the scope open to adding slist.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
On Thu, 3 Sept 2026 at 00:48, David Rowley <dgrowleyml@gmail.com> wrote:
On Wed, 2 Sept 2026 at 23:36, Matthias van de Meent
<boekewurm+postgres@gmail.com> wrote:Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].This feels like putting the cart before the horse. What purpose does
maintaining a count in the dlist serve at all for [1]?
To check the consistency of the dlist, and with it, the consistency of
this part of the memory context's overall structure? It's the same
reason we have chunk->requested_size and checks to validate its
consistency with other information, and check the sum of allocated
memory against contex->mem_allocated in practically every mcm.check
implementation. Dlists don't inherently have an authorative indicator
of where every member exists or how many there are, and therefore
don't have much corruption protection if something goes wrong; all you
have is a progressively explored list of elements that you hope is
consistent and loops back to the list head.
Last I looked
at the 0004 patch on [1] you were only counting and asserting the
count in debug builds. Since the count is not used for anything else,
it seemed to me that all the assert was doing was verifying the count
tracking was correct, and since that count tracking code only existed
in debug builds, there was no point in it.
It makes a lot of sense to me to make sure that a linked list (when
that is critical to your data structure's integrity) has not
accidentally lost entries or received new ones unexpectedly. I don't
think it's much different from checking that the sum total of your
context's allocations adds up to context->mem_allocated, and that the
sentinel bytes haven't been overwritten. Yes, the condition should be
invariantly correct, but you prefer knowing that something was wrong
if that invariant was broken somehow.
And yes, debug builds are exactly where you add deep validation to
memory contexts, as that is when MCM->check is used and available.
I think if you have a concrete proposal for your 0001 patch here,
there should be a 0002 patch which uses the new macros for some
legitimate reason. Otherwise, if the only legit reason is in [1], then
this patch should be part of that series. At the moment, all this
thread proposes to do is introduce dead code.
Every new API needs a first caller, sure. But in this case, right now,
I can't find _any_ uses of _Generic in our current headers, apart from
StaticAssertVariableIsOfType*, and a MSVC-specific macro for
pg_integer_constant_p. No real APIs are currently exposed with
_Generic -wrapping macros, hence why I started a separate thread to
accumulate interest and feedback on using this new C11 feature to
provide a simple unified API against two implementations that
developers may want to select between.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
On Wed, 02 Sep 2026 at 17:59, Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:
On Wed, 2 Sept 2026 at 17:32, Japin Li <japinli@hotmail.com> wrote:
On Wed, 02 Sep 2026 at 13:36, Matthias van de Meent <boekewurm+postgres@gmail.com> wrote:
Hi,
I'd like to track the size of a dlist for data structure validation
purposes[^0]. Normally, one would use a dclist, as this tracks a
count of list elements contained therein, but because this code would
not be called in most normal production builds using a dclist would
waste precious memory.
Manually tracking the length is possible, but tedious, and a local
wrapper around the used dclist/dlist APIs (with different
implementations conditioned with #ifdefs to use the right types) would
also be a significant amount of effort, that'd be duplicated every
time .Attached is a patch that adds glist_* macros, which wrap several
dlist/dclist_* APIs, so that developers can use dlist/dclist
selectively in different environments, without significant visual
overhead in the code. I'm planning to use this in the Proxy memory
contexts over at [1].I've considered also adding slist_* to the macros, but I've never
needed selective slist vs dlist/dclist before, so I ignored that list
type for now.+1 for this idea.
Should we adopt the new glist interface in the existing code to better
showcase its intended use?Did you have a place in mind where we have dlist and dclist in use to
track the same information in different build configurations?
I don't have such a place in mind.
I don't
like churning code for the sake of new APIs, especially when those new
APIs are not strictly better than the current ones.Note that I add glist to solve ergonomics issues when you want to
handle both types of lists. It's not meant to be used as default API
to fields that are always of a dlist type, or always of a dclist type,
even if it could be used like that.
If there is no code that actually uses it, I don't see a justification for
adding it, especially given the concern about code churn.
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.