Adding doubly linked list type which stores the number of items in the list
As part of the AIO work [1]/messages/by-id/20210223100344.llw5an2aklengrmn@alap3.anarazel.de, there are quite a number of dlist_heads
which a counter is used to keep track on how many items are in the
list. We also have a few places in master which do the same thing.
In order to tidy this up and to help ensure that the count variable
does not get out of sync with the items which are stored in the list,
how about we introduce "dclist" which maintains the count for us?
I've attached a patch which does this. The majority of the functions
for the new type are just wrappers around the equivalent dlist
function.
dclist provides all of the functionality that dlist does except
there's no dclist_delete() function. dlist_delete() can be done by
just knowing the element to delete and not the list that the element
belongs to. With dclist, that's not possible as we must also subtract
1 from the count variable and obviously we need the dclist_head for
that.
I'll add this to the November commitfest.
David
[1]: /messages/by-id/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
Attachments:
v1-0001-Add-doubly-linked-count-list-implementation.patchtext/plain; charset=US-ASCII; name=v1-0001-Add-doubly-linked-count-list-implementation.patchDownload
From a24babfe52c7ab946f192f4872dc480ce18c8c0f Mon Sep 17 00:00:00 2001
From: David Rowley <dgrowley@gmail.com>
Date: Thu, 27 Oct 2022 16:01:34 +1300
Subject: [PATCH v1] Add doubly linked count list implementation
We have various requirements when using a dlist_head to keep track of the
number of items in the list. This, traditionally, has been done by
maintaining a counter variable in the calling code. Here we tidy this up
by adding "dclist", which is very similar to dlists but also keeps track
of the number of items stored in the list.
Callers may use the new dclist_count() function when they need to know how
many items are stored. Obtaining the count is an O(1) operation.
For simplicity reasons, dclists and dlists both use dlist_node as their
node type and dlist_iter/dlist_mutable_iter as their iterator type.
dclists have all of the same functionality as dlists except there is no
function named dclist_delete(). To remove an item from a list
dclist_delete_from() must be used. This requires knowing which dclist the
given item is stored in.
Additionally, here we also convert some dlists which keep track of the
number of items stored to make them use dclists instead.
---
.../replication/logical/reorderbuffer.c | 58 ++--
src/backend/replication/logical/snapbuild.c | 2 +-
src/backend/utils/activity/pgstat_xact.c | 34 +-
src/backend/utils/adt/ri_triggers.c | 13 +-
src/include/lib/ilist.h | 313 +++++++++++++++++-
src/include/replication/reorderbuffer.h | 6 +-
src/include/utils/pgstat_internal.h | 3 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 351 insertions(+), 79 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c29894041e..603c861461 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -349,8 +349,6 @@ ReorderBufferAllocate(void)
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
- buffer->catchange_ntxns = 0;
-
buffer->outbuf = NULL;
buffer->outbufsize = 0;
buffer->size = 0;
@@ -368,7 +366,7 @@ ReorderBufferAllocate(void)
dlist_init(&buffer->toplevel_by_lsn);
dlist_init(&buffer->txns_by_base_snapshot_lsn);
- dlist_init(&buffer->catchange_txns);
+ dclist_init(&buffer->catchange_txns);
/*
* Ensure there's no stale data from prior uses of this slot, in case some
@@ -413,7 +411,7 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->changes);
dlist_init(&txn->tuplecids);
- dlist_init(&txn->subtxns);
+ dclist_init(&txn->subtxns);
/* InvalidCommandId is not zero, so set it explicitly */
txn->command_id = InvalidCommandId;
@@ -1066,14 +1064,13 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
- Assert(subtxn->nsubtxns == 0);
+ Assert(dclist_count(&subtxn->subtxns) == 0);
/* set the reference to top-level transaction */
subtxn->toptxn = txn;
/* add to subtransaction list */
- dlist_push_tail(&txn->subtxns, &subtxn->node);
- txn->nsubtxns++;
+ dclist_push_tail(&txn->subtxns, &subtxn->node);
/* Possibly transfer the subtxn's snapshot to its top-level txn. */
ReorderBufferTransferSnapToParent(txn, subtxn);
@@ -1241,7 +1238,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->nentries > 0)
nr_txns++;
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ dclist_foreach(cur_txn_i, &txn->subtxns)
{
ReorderBufferTXN *cur_txn;
@@ -1308,7 +1305,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ dclist_foreach(cur_txn_i, &txn->subtxns)
{
ReorderBufferTXN *cur_txn;
@@ -1477,7 +1474,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_mutable_iter iter;
/* cleanup subtransactions & their changes */
- dlist_foreach_modify(iter, &txn->subtxns)
+ dclist_foreach_modify(iter, &txn->subtxns)
{
ReorderBufferTXN *subtxn;
@@ -1489,7 +1486,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* ever recurse more than one level deep here.
*/
Assert(rbtxn_is_known_subxact(subtxn));
- Assert(subtxn->nsubtxns == 0);
+ Assert(dclist_count(&subtxn->subtxns) == 0);
ReorderBufferCleanupTXN(rb, subtxn);
}
@@ -1546,19 +1543,14 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* Remove TXN from its containing lists.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
- * parent's list of known subxacts; this leaves the parent's nsubxacts
+ * parent's list of known subxacts; this leaves the parent's subxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs. We remove the TXN from the
* list of catalog modifying transactions as well.
*/
dlist_delete(&txn->node);
if (rbtxn_has_catalog_changes(txn))
- {
- dlist_delete(&txn->catchange_node);
- rb->catchange_ntxns--;
-
- Assert(rb->catchange_ntxns >= 0);
- }
+ dclist_delete_from(&rb->catchange_txns, &txn->catchange_node);
/* now remove reference from buffer */
hash_search(rb->by_txn,
@@ -1592,7 +1584,7 @@ ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn, bool txn_prep
dlist_mutable_iter iter;
/* cleanup subtransactions & their changes */
- dlist_foreach_modify(iter, &txn->subtxns)
+ dclist_foreach_modify(iter, &txn->subtxns)
{
ReorderBufferTXN *subtxn;
@@ -1604,7 +1596,7 @@ ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn, bool txn_prep
* ever recurse more than one level deep here.
*/
Assert(rbtxn_is_known_subxact(subtxn));
- Assert(subtxn->nsubtxns == 0);
+ Assert(dclist_count(&subtxn->subtxns) == 0);
ReorderBufferTruncateTXN(rb, subtxn, txn_prepared);
}
@@ -1788,7 +1780,7 @@ ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
size = sizeof(SnapshotData) +
sizeof(TransactionId) * orig_snap->xcnt +
- sizeof(TransactionId) * (txn->nsubtxns + 1);
+ sizeof(TransactionId) * (dclist_count(&txn->subtxns) + 1);
snap = MemoryContextAllocZero(rb->context, size);
memcpy(snap, orig_snap, sizeof(SnapshotData));
@@ -1815,7 +1807,7 @@ ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
*/
snap->subxcnt = 1;
- dlist_foreach(iter, &txn->subtxns)
+ dclist_foreach(iter, &txn->subtxns)
{
ReorderBufferTXN *sub_txn;
@@ -3309,8 +3301,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (!rbtxn_has_catalog_changes(txn))
{
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &txn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &txn->catchange_node);
}
/*
@@ -3323,8 +3314,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (toptxn != NULL && !rbtxn_has_catalog_changes(toptxn))
{
toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
}
}
@@ -3342,15 +3332,13 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
size_t xcnt = 0;
/* Quick return if the list is empty */
- if (rb->catchange_ntxns == 0)
- {
- Assert(dlist_is_empty(&rb->catchange_txns));
+ if (dclist_count(&rb->catchange_txns) == 0)
return NULL;
- }
/* Initialize XID array */
- xids = (TransactionId *) palloc(sizeof(TransactionId) * rb->catchange_ntxns);
- dlist_foreach(iter, &rb->catchange_txns)
+ xids = (TransactionId *) palloc(sizeof(TransactionId) *
+ dclist_count(&rb->catchange_txns));
+ dclist_foreach(iter, &rb->catchange_txns)
{
ReorderBufferTXN *txn = dlist_container(ReorderBufferTXN,
catchange_node,
@@ -3363,7 +3351,7 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
qsort(xids, xcnt, sizeof(TransactionId), xidComparator);
- Assert(xcnt == rb->catchange_ntxns);
+ Assert(xcnt == dclist_count(&rb->catchange_txns));
return xids;
}
@@ -3610,7 +3598,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
(uint32) txn->nentries_mem, txn->xid);
/* do the same to all child TXs */
- dlist_foreach(subtxn_i, &txn->subtxns)
+ dclist_foreach(subtxn_i, &txn->subtxns)
{
ReorderBufferTXN *subtxn;
@@ -3975,7 +3963,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* at the beginning we should have invalid command ID */
Assert(txn->command_id == InvalidCommandId);
- dlist_foreach(subxact_i, &txn->subtxns)
+ dclist_foreach(subxact_i, &txn->subtxns)
{
ReorderBufferTXN *subtxn;
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index f892d19bfb..5006a5c464 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1688,7 +1688,7 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
/* Get the catalog modifying transactions that are yet not committed */
catchange_xip = ReorderBufferGetCatalogChangesXacts(builder->reorder);
- catchange_xcnt = builder->reorder->catchange_ntxns;
+ catchange_xcnt = dclist_count(&builder->reorder->catchange_txns);
needed_length = sizeof(SnapBuildOnDisk) +
sizeof(TransactionId) * (builder->committed.xcnt + catchange_xcnt);
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index d6f660edf7..d5284cfbde 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -70,13 +70,10 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
- {
- Assert(dlist_is_empty(&xact_state->pending_drops));
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
- }
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
@@ -101,8 +98,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
not_freed_count++;
}
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
pfree(pending);
}
@@ -144,19 +140,18 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
parent_xact_state = pgstat_get_xact_stack_level(nestDepth - 1);
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
xl_xact_stats_item *it = &pending->item;
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
if (!isCommit && pending->is_create)
{
@@ -175,8 +170,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
* remove the stats object, the surrounding transaction might
* still abort. Pass it on to the parent.
*/
- dlist_push_tail(&parent_xact_state->pending_drops, &pending->node);
- parent_xact_state->pending_drops_count++;
+ dclist_push_tail(&parent_xact_state->pending_drops, &pending->node);
}
else
{
@@ -184,7 +178,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
}
}
- Assert(xact_state->pending_drops_count == 0);
+ Assert(dclist_count(&xact_state->pending_drops) == 0);
if (not_freed_count > 0)
pgstat_request_entry_refs_gc();
}
@@ -250,8 +244,7 @@ pgstat_get_xact_stack_level(int nest_level)
xact_state = (PgStat_SubXactStatus *)
MemoryContextAlloc(TopTransactionContext,
sizeof(PgStat_SubXactStatus));
- dlist_init(&xact_state->pending_drops);
- xact_state->pending_drops_count = 0;
+ dclist_init(&xact_state->pending_drops);
xact_state->nest_level = nest_level;
xact_state->prev = pgStatXactStack;
xact_state->first = NULL;
@@ -291,10 +284,10 @@ pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items)
Assert(!isCommit || xact_state->nest_level == 1);
Assert(!isCommit || xact_state->prev == NULL);
- *items = palloc(xact_state->pending_drops_count
+ *items = palloc(dclist_count(&xact_state->pending_drops)
* sizeof(xl_xact_stats_item));
- dlist_foreach(iter, &xact_state->pending_drops)
+ dclist_foreach(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
@@ -304,7 +297,7 @@ pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items)
if (!isCommit && !pending->is_create)
continue;
- Assert(nitems < xact_state->pending_drops_count);
+ Assert(nitems < dclist_count(&xact_state->pending_drops));
(*items)[nitems++] = pending->item;
}
@@ -351,8 +344,7 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, Oid objoid, bool
drop->item.dboid = dboid;
drop->item.objoid = objoid;
- dlist_push_tail(&xact_state->pending_drops, &drop->node);
- xact_state->pending_drops_count++;
+ dclist_push_tail(&xact_state->pending_drops, &drop->node);
}
/*
diff --git a/src/backend/utils/adt/ri_triggers.c b/src/backend/utils/adt/ri_triggers.c
index 1d503e7e01..25c09dc102 100644
--- a/src/backend/utils/adt/ri_triggers.c
+++ b/src/backend/utils/adt/ri_triggers.c
@@ -176,8 +176,7 @@ typedef struct RI_CompareHashEntry
static HTAB *ri_constraint_cache = NULL;
static HTAB *ri_query_cache = NULL;
static HTAB *ri_compare_cache = NULL;
-static dlist_head ri_constraint_cache_valid_list;
-static int ri_constraint_cache_valid_count = 0;
+static dclist_head ri_constraint_cache_valid_list;
/*
@@ -2174,8 +2173,7 @@ ri_LoadConstraintInfo(Oid constraintOid)
* For efficient processing of invalidation messages below, we keep a
* doubly-linked list, and a count, of all currently valid entries.
*/
- dlist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
- ri_constraint_cache_valid_count++;
+ dclist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
riinfo->valid = true;
@@ -2233,10 +2231,10 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
* O(N^2) behavior in situations where a session touches many foreign keys
* and also does many ALTER TABLEs, such as a restore from pg_dump.
*/
- if (ri_constraint_cache_valid_count > 1000)
+ if (dclist_count(&ri_constraint_cache_valid_list) > 1000)
hashvalue = 0; /* pretend it's a cache reset */
- dlist_foreach_modify(iter, &ri_constraint_cache_valid_list)
+ dclist_foreach_modify(iter, &ri_constraint_cache_valid_list)
{
RI_ConstraintInfo *riinfo = dlist_container(RI_ConstraintInfo,
valid_link, iter.cur);
@@ -2252,8 +2250,7 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
{
riinfo->valid = false;
/* Remove invalidated entries from the list, too */
- dlist_delete(iter.cur);
- ri_constraint_cache_valid_count--;
+ dclist_delete_from(&ri_constraint_cache_valid_list, iter.cur);
}
}
}
diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index 7ab0888f4f..221f7aac42 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -10,20 +10,33 @@
* the link fields in the remainder would be wasted space. But usually,
* it saves space to not have separately-allocated list nodes.)
*
+ * The doubly-linked list comes in 2 forms. dlist_head defines a head of a
+ * doubly-linked list of dlist_nodes. Whereas dclist_head defines the head of
+ * a doubly-linked list of dlist_nodes with an additional 'count' field to
+ * keep track of how many items are contained within the given list. For
+ * simplicity dlist_head and dclist_head share the same node type and
+ * iterator types. Each function to manipulate a dlist_head is prefixed by
+ * dlist, whereas functions to manipulate a dclist_head are prefixed with
+ * dclist. dclist_head comes with an additional function (dclist_count) to
+ * return the number of entries in the list. dclists are able to store a
+ * maximum of UINT32 elements. It is up to the caller to ensure no more than
+ * this many items are added to a dclist.
+ *
* None of the functions here allocate any memory; they just manipulate
* externally managed memory. The APIs for singly and doubly linked lists
* are identical as far as capabilities of both allow.
*
* Each list has a list header, which exists even when the list is empty.
* An empty singly-linked list has a NULL pointer in its header.
- * There are two kinds of empty doubly linked lists: those that have been
- * initialized to NULL, and those that have been initialized to circularity.
- * (If a dlist is modified and then all its elements are deleted, it will be
- * in the circular state.) We prefer circular dlists because there are some
- * operations that can be done without branches (and thus faster) on lists
- * that use circular representation. However, it is often convenient to
- * initialize list headers to zeroes rather than setting them up with an
- * explicit initialization function, so we also allow the other case.
+ * For both dlist_head and dclist_head there are two kinds of empty doubly
+ * linked lists: those that have been initialized to NULL, and those that have
+ * been initialized to circularity. (If a dlist is modified and then all its
+ * elements are deleted, it will be in the circular state.) We prefer circular
+ * dlists because there are some operations that can be done without branches
+ * (and thus faster) on lists that use circular representation. However, it
+ * is often convenient to initialize list headers to zeroes rather than
+ * setting them up with an explicit initialization function, so we also allow
+ * the other case.
*
* EXAMPLES
*
@@ -182,6 +195,19 @@ typedef struct dlist_mutable_iter
dlist_node *end; /* last node we'll iterate to */
} dlist_mutable_iter;
+/*
+ * Head of a doubly linked list with a count of the number of items
+ *
+ * This internally makes use of a dlist to implement the actual list and
+ * wrapper functions are used to maintain the count when items are added and
+ * removed from the list.
+ */
+typedef struct dclist_head
+{
+ dlist_head dlist; /* the actual list header */
+ uint32 count; /* the number of items in the list */
+} dclist_head;
+
/*
* Node of a singly linked list.
*
@@ -246,6 +272,7 @@ typedef struct slist_mutable_iter
/* Static initializers */
#define DLIST_STATIC_INIT(name) {{&(name).head, &(name).head}}
+#define DCLIST_STATIC_INIT(name) {{&(name).head, &(name).head}, 0}
#define SLIST_STATIC_INIT(name) {{NULL}}
@@ -270,6 +297,23 @@ extern void slist_check(slist_head *head);
/* doubly linked list implementation */
+#ifdef ILIST_DEBUG
+/*
+ * dlist_is_memberof
+ * Returns true if 'node' is a member of 'head'.
+ */
+static bool
+dlist_is_memberof(dlist_node *node, dlist_head *head)
+{
+ for (dlist_node *cur = &head->head; cur != NULL; cur = cur->next)
+ {
+ if (cur == node)
+ return true;
+ }
+ return false;
+}
+#endif
+
/*
* Initialize a doubly linked list.
* Previous state will be thrown away without any cleanup.
@@ -361,6 +405,19 @@ dlist_delete(dlist_node *node)
node->next->prev = node->prev;
}
+/*
+ * As dlist_delete but performs checks in ILIST_DEBUG to ensure that 'node'
+ * belongs to 'head'.
+ */
+static inline void
+dlist_delete_from(dlist_head *head, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(node, head));
+#endif
+ dlist_delete(node);
+}
+
/*
* Remove and return the first node from a list (there must be one).
*/
@@ -562,6 +619,246 @@ dlist_tail_node(dlist_head *head)
(iter).cur != (iter).end; \
(iter).cur = (iter).cur->prev)
+/* double linked list with count implementation */
+
+/*
+ * Initialize a doubly linked count list.
+ * Previous state will be thrown away without any cleanup.
+ */
+static inline void
+dclist_init(dclist_head *head)
+{
+ dlist_init(&head->dlist);
+ head->count = 0;
+}
+
+/*
+ * dclist_is_empty
+ * Returns true if the list is empty, otherwise false.
+ */
+static inline bool
+dclist_is_empty(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+ return dlist_is_empty(&head->dlist);
+}
+
+/*
+ * dclist_push_head
+ * Insert a node at the beginning of the list.
+ */
+static inline void
+dclist_push_head(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_head(&head->dlist, node);
+ head->count++;
+}
+
+/*
+ * dclist_push_tail
+ * Insert a node at the end of the list.
+ */
+static inline void
+dclist_push_tail(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_tail(&head->dlist, node);
+ head->count++;
+}
+
+/*
+ * dclist_insert_after
+ * Insert a node after another *in the same list*
+ *
+ * Caller must ensure that 'after' is a member of 'head'.
+ */
+static inline void
+dclist_insert_after(dclist_head *head, dlist_node *after, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(after, &head->dlist));
+#endif
+
+ dlist_insert_after(after, node);
+ head->count++;
+}
+
+/*
+ * dclist_insert_before
+ * Insert a node before another *in the same list*
+ *
+ * Caller must ensure that 'before' is a member of 'head'.
+ */
+static inline void
+dclist_insert_before(dclist_head *head, dlist_node *before, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(before, &head->dlist));
+#endif
+
+ dlist_insert_before(before, node);
+ head->count++;
+}
+
+/*
+ * dclist_delete_from
+ * Deletes 'node' from 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_delete_from(dclist_head *head, dlist_node *node)
+{
+ dlist_delete_from(&head->dlist, node);
+ head->count--;
+}
+
+/*
+ * dclist_pop_head_node
+ * Remove and return the first node from a list (there must be one).
+ */
+static inline dlist_node *
+dclist_pop_head_node(dclist_head *head)
+{
+ dlist_node *node;
+
+ Assert(head->count > 0);
+
+ node = dlist_pop_head_node(&head->dlist);
+ head->count--;
+
+ return node;
+}
+
+/*
+ * dclist_move_head
+ * Move 'node' from its current position in the list to the head position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_head(dclist_head *head, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(node, &head->dlist));
+#endif
+
+ dlist_move_head(&head->dlist, node);
+}
+
+/*
+ * dclist_move_tail
+ * Move 'node' from its current position in the list to the tail position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_tail(dclist_head *head, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(node, &head->dlist));
+#endif
+
+ dlist_move_tail(&head->dlist, node);
+}
+
+/*
+ * dclist_has_next
+ * Check whether 'node' has a following node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_next(dclist_head *head, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(node, &head->dlist));
+#endif
+
+ return dlist_has_next(&head->dlist, node);
+}
+
+/*
+ * dclist_has_prev
+ * Check whether 'node' has a preceding node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_prev(dclist_head *head, dlist_node *node)
+{
+#ifdef ILIST_DEBUG
+ Assert(dlist_is_memberof(node, &head->dlist));
+#endif
+
+ return dlist_has_prev(&head->dlist, node);
+}
+
+/*
+ * dclist_next_node
+ * Return the next node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_next_node(dclist_head *head, dlist_node *node)
+{
+ return dlist_next_node(&head->dlist, node);
+}
+
+/*
+ * dclist_prev_node
+ * Return the prev node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_prev_node(dclist_head *head, dlist_node *node)
+{
+ return dlist_prev_node(&head->dlist, node);
+}
+
+/*
+ * dclist_head_node
+ * Return the first node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_head_node(dclist_head *head)
+{
+ return (dlist_node *) dlist_head_element_off(&head->dlist, 0);
+}
+
+/*
+ * Return the last node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_tail_node(dclist_head *head)
+{
+ return (dlist_node *) dlist_tail_element_off(&head->dlist, 0);
+}
+
+/*
+ * dclist_count
+ * Returns the stored number of entries in 'head'
+ */
+static inline uint32
+dclist_count(dclist_head *head)
+{
+ return head->count;
+}
+
+/* Iterators for dclists */
+#define dclist_foreach(iter, lhead) \
+ dlist_foreach(iter, &((lhead)->dlist))
+
+#define dclist_foreach_modify(iter, lhead) \
+ dlist_foreach_modify(iter, &((lhead)->dlist))
+
+#define dclist_reverse_foreach(iter, lhead) \
+ dlist_reverse_foreach(iter, &((lhead)->dlist))
/* singly linked list implementation */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 02b59a1931..7301555b89 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -361,8 +361,7 @@ typedef struct ReorderBufferTXN
* non-hierarchical list of subtransactions that are *not* aborted. Only
* used in toplevel transactions.
*/
- dlist_head subtxns;
- uint32 nsubtxns;
+ dclist_head subtxns;
/*
* Stored cache invalidations. This is not a linked list because we get
@@ -534,8 +533,7 @@ struct ReorderBuffer
/*
* Transactions and subtransactions that have modified system catalogs.
*/
- dlist_head catchange_txns;
- int catchange_ntxns;
+ dclist_head catchange_txns;
/*
* one-entry sized cache for by_txn. Very frequently the same txn gets
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 627c1389e4..9f32a6f6d3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -162,8 +162,7 @@ typedef struct PgStat_SubXactStatus
* if the transaction commits/aborts. To handle replicas and crashes,
* stats drops are included in commit / abort records.
*/
- dlist_head pending_drops;
- int pending_drops_count;
+ dclist_head pending_drops;
/*
* Tuple insertion/deletion counts for an open transaction can't be
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2f02cc8f42..c15f990cbb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3194,6 +3194,7 @@ datapagemap_t
dateKEY
datetkn
dce_uuid_t
+dclist_head
decimal
deparse_columns
deparse_context
--
2.34.1
On Thu, Oct 27, 2022 at 9:05 AM David Rowley <dgrowleyml@gmail.com> wrote:
As part of the AIO work [1], there are quite a number of dlist_heads
which a counter is used to keep track on how many items are in the
list. We also have a few places in master which do the same thing.In order to tidy this up and to help ensure that the count variable
does not get out of sync with the items which are stored in the list,
how about we introduce "dclist" which maintains the count for us?I've attached a patch which does this. The majority of the functions
for the new type are just wrappers around the equivalent dlist
function.dclist provides all of the functionality that dlist does except
there's no dclist_delete() function. dlist_delete() can be done by
just knowing the element to delete and not the list that the element
belongs to. With dclist, that's not possible as we must also subtract
1 from the count variable and obviously we need the dclist_head for
that.[1] /messages/by-id/20210223100344.llw5an2aklengrmn@alap3.anarazel.de
+1. Using dlist_head in dclist_head enables us to reuse dlist_* functions.
Some comments on the patch:
1. I think it's better to just return dlist_is_empty(&head->dlist) &&
(head->count == 0); from dclist_is_empty() and remove the assert for
better readability and safety against count being zero.
2. Missing dlist_is_memberof() in dclist_delete_from()?
3. Just thinking if we need to move dlist_is_memberof() check from
dclist_* functions to dlist_* functions, because they also need such
insurance against callers passing spurious nodes.
4. More opportunities to use dclist_* in below places, no?
dlist_push_tail(&src->mappings, &pmap->node);
src->num_mappings++;
dlist_push_head(&MXactCache, &entry->node);
if (MXactCacheMembers++ >= MAX_CACHE_ENTRIES)
5. dlist_is_memberof() - do we need this at all? We trust the callers
of dlist_* today that the passed in node belongs to the list, no?
6. If we decide to have dlist_is_memberof() and the asserts around it,
can we design it on the similar lines as dlist_check() to avoid many
#ifdef ILIST_DEBUG #endif blocks spreading around the code?
7. Do we need Assert(head->count > 0); in more places like dclist_delete_from()?
8. Don't we need dlist_container(), dlist_head_element(),
dlist_tail_element() for dclist_*? Even though, we might not use them
immediately, just for the sake for completeness of dclist data
structure.
I'll add this to the November commitfest.
Yes, please.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Thank you for having a look at this.
On Thu, 27 Oct 2022 at 19:32, Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
Some comments on the patch:
1. I think it's better to just return dlist_is_empty(&head->dlist) &&
(head->count == 0); from dclist_is_empty() and remove the assert for
better readability and safety against count being zero.
I don't think that's a good change. For 1) it adds unnecessary
overhead due to the redundant checks and 2) it removes the Assert
which is our early warning that the dclist's count is getting out of
sync somewhere.
2. Missing dlist_is_memberof() in dclist_delete_from()?
I put that in dlist_delete_from() which is called from dclist_delete_from().
3. Just thinking if we need to move dlist_is_memberof() check from
dclist_* functions to dlist_* functions, because they also need such
insurance against callers passing spurious nodes.
I think the affected functions there would be; dlist_move_head(),
dlist_move_tail(), dlist_has_next(), dlist_has_prev(),
dlist_next_node() and dlist_prev_node(). I believe if we did that then
it's effectively an API change. The comments only claim that it's
undefined if node is not a member of the list. It does not say 'node'
*must* be part of the list. Now, perhaps doing this would just make
it more likely that we'd find bugs in our code and extension authors
would find bugs in their code, but it does move the bar.
dlist_move_head and dlist_move_tail look like they'd work perfectly
well to remove an item from 1 list and put it on the head or tail of
some completely different list. Should we really be changing that in a
patch that is meant to just add the dclist type?
4. More opportunities to use dclist_* in below places, no?
dlist_push_tail(&src->mappings, &pmap->node);
src->num_mappings++;dlist_push_head(&MXactCache, &entry->node);
if (MXactCacheMembers++ >= MAX_CACHE_ENTRIES)
Thanks for finding those. I've adjusted them both to use dclists.
5. dlist_is_memberof() - do we need this at all? We trust the callers
of dlist_* today that the passed in node belongs to the list, no?
hmm, this seems to contradict your #3?
If you look at something like dlist_move_head(), if someone calls that
and passes a 'node' that does not belong to 'head' then the result of
that is that we delete 'node' from whichever dlist that it's on and
push it onto 'head'. Nothing bad happens there. If we do the same on
a dclist then the count gets out of sync. That's bad as it could lead
to assert failures and bugs.
6. If we decide to have dlist_is_memberof() and the asserts around it,
can we design it on the similar lines as dlist_check() to avoid many
#ifdef ILIST_DEBUG #endif blocks spreading around the code?
OK, that likely is a better idea. I've done this in the attached by
way of dlist_member_check()
7. Do we need Assert(head->count > 0); in more places like dclist_delete_from()?
I guess it does no harm. I've added some additional ones in the attached.
8. Don't we need dlist_container(), dlist_head_element(),
dlist_tail_element() for dclist_*? Even though, we might not use them
immediately, just for the sake for completeness of dclist data
structure.
OK, I think I'd left those because dclist_container() would just be
the same as dlist_container(), but that's not the case for the other
two, so I've added all 3.
One additional change is that I also ended up removing the use of
dclist that I had in the previous patch for ReorderBufferTXN.subtxns.
Looking more closely at the code in ReorderBufferAssignChild():
/*
* We already saw this transaction, but initially added it to the
* list of top-level txns. Now that we know it's not top-level,
* remove it from there.
*/
dlist_delete(&subtxn->node);
The problem is that since ReorderBufferTXN is used for both
transactions and sub-transactions that it's not easy to determine if
the ReorderBufferTXN.node is part of the ReorderBuffer.toplevel_by_lsn
dlist or the ReorderBufferTXN.subtxns. It seems safer just to leave
this one alone.
David
Attachments:
v2_add_doubly_linked_count_lists.patchapplication/x-patch; name=v2_add_doubly_linked_count_lists.patchDownload
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index b01b39b008..a34e9b352d 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -196,8 +196,7 @@ typedef struct RewriteMappingFile
TransactionId xid; /* xid that might need to see the row */
int vfd; /* fd of mappings file */
off_t off; /* how far have we written yet */
- uint32 num_mappings; /* number of in-memory mappings */
- dlist_head mappings; /* list of in-memory mappings */
+ dclist_head mappings; /* list of in-memory mappings */
char path[MAXPGPATH]; /* path, for error messages */
} RewriteMappingFile;
@@ -864,9 +863,10 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
Oid dboid;
uint32 len;
int written;
+ uint32 num_mappings = dclist_count(&src->mappings);
/* this file hasn't got any new mappings */
- if (src->num_mappings == 0)
+ if (num_mappings == 0)
continue;
if (state->rs_old_rel->rd_rel->relisshared)
@@ -874,7 +874,7 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
else
dboid = MyDatabaseId;
- xlrec.num_mappings = src->num_mappings;
+ xlrec.num_mappings = num_mappings;
xlrec.mapped_rel = RelationGetRelid(state->rs_old_rel);
xlrec.mapped_xid = src->xid;
xlrec.mapped_db = dboid;
@@ -882,31 +882,30 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
xlrec.start_lsn = state->rs_begin_lsn;
/* write all mappings consecutively */
- len = src->num_mappings * sizeof(LogicalRewriteMappingData);
+ len = num_mappings * sizeof(LogicalRewriteMappingData);
waldata_start = waldata = palloc(len);
/*
* collect data we need to write out, but don't modify ondisk data yet
*/
- dlist_foreach_modify(iter, &src->mappings)
+ dclist_foreach_modify(iter, &src->mappings)
{
RewriteMappingDataEntry *pmap;
- pmap = dlist_container(RewriteMappingDataEntry, node, iter.cur);
+ pmap = dclist_container(RewriteMappingDataEntry, node, iter.cur);
memcpy(waldata, &pmap->map, sizeof(pmap->map));
waldata += sizeof(pmap->map);
/* remove from the list and free */
- dlist_delete(&pmap->node);
+ dclist_delete_from(&src->mappings, &pmap->node);
pfree(pmap);
/* update bookkeeping */
state->rs_num_rewrite_mappings--;
- src->num_mappings--;
}
- Assert(src->num_mappings == 0);
+ Assert(dclist_count(&src->mappings) == 0);
Assert(waldata == waldata_start + len);
/*
@@ -1002,8 +1001,7 @@ logical_rewrite_log_mapping(RewriteState state, TransactionId xid,
LSN_FORMAT_ARGS(state->rs_begin_lsn),
xid, GetCurrentTransactionId());
- dlist_init(&src->mappings);
- src->num_mappings = 0;
+ dclist_init(&src->mappings);
src->off = 0;
memcpy(src->path, path, sizeof(path));
src->vfd = PathNameOpenFile(path,
@@ -1017,8 +1015,7 @@ logical_rewrite_log_mapping(RewriteState state, TransactionId xid,
pmap = MemoryContextAlloc(state->rs_cxt,
sizeof(RewriteMappingDataEntry));
memcpy(&pmap->map, map, sizeof(LogicalRewriteMappingData));
- dlist_push_tail(&src->mappings, &pmap->node);
- src->num_mappings++;
+ dclist_push_tail(&src->mappings, &pmap->node);
state->rs_num_rewrite_mappings++;
/*
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index a7383f553b..a03749061b 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -319,8 +319,7 @@ typedef struct mXactCacheEnt
} mXactCacheEnt;
#define MAX_CACHE_ENTRIES 256
-static dlist_head MXactCache = DLIST_STATIC_INIT(MXactCache);
-static int MXactCacheMembers = 0;
+static dclist_head MXactCache = DCLIST_STATIC_INIT(MXactCache);
static MemoryContext MXactContext = NULL;
#ifdef MULTIXACT_DEBUG
@@ -1504,9 +1503,9 @@ mXactCacheGetBySet(int nmembers, MultiXactMember *members)
/* sort the array so comparison is easy */
qsort(members, nmembers, sizeof(MultiXactMember), mxactMemberComparator);
- dlist_foreach(iter, &MXactCache)
+ dclist_foreach(iter, &MXactCache)
{
- mXactCacheEnt *entry = dlist_container(mXactCacheEnt, node, iter.cur);
+ mXactCacheEnt *entry = dclist_container(mXactCacheEnt, node, iter.cur);
if (entry->nmembers != nmembers)
continue;
@@ -1518,7 +1517,7 @@ mXactCacheGetBySet(int nmembers, MultiXactMember *members)
if (memcmp(members, entry->members, nmembers * sizeof(MultiXactMember)) == 0)
{
debug_elog3(DEBUG2, "CacheGet: found %u", entry->multi);
- dlist_move_head(&MXactCache, iter.cur);
+ dclist_move_head(&MXactCache, iter.cur);
return entry->multi;
}
}
@@ -1542,9 +1541,9 @@ mXactCacheGetById(MultiXactId multi, MultiXactMember **members)
debug_elog3(DEBUG2, "CacheGet: looking for %u", multi);
- dlist_foreach(iter, &MXactCache)
+ dclist_foreach(iter, &MXactCache)
{
- mXactCacheEnt *entry = dlist_container(mXactCacheEnt, node, iter.cur);
+ mXactCacheEnt *entry = dclist_container(mXactCacheEnt, node, iter.cur);
if (entry->multi == multi)
{
@@ -1566,7 +1565,7 @@ mXactCacheGetById(MultiXactId multi, MultiXactMember **members)
* This is acceptable only because we exit the iteration
* immediately afterwards.
*/
- dlist_move_head(&MXactCache, iter.cur);
+ dclist_move_head(&MXactCache, iter.cur);
*members = ptr;
return entry->nmembers;
@@ -1610,14 +1609,13 @@ mXactCachePut(MultiXactId multi, int nmembers, MultiXactMember *members)
/* mXactCacheGetBySet assumes the entries are sorted, so sort them */
qsort(entry->members, nmembers, sizeof(MultiXactMember), mxactMemberComparator);
- dlist_push_head(&MXactCache, &entry->node);
- if (MXactCacheMembers++ >= MAX_CACHE_ENTRIES)
+ dclist_push_head(&MXactCache, &entry->node);
+ if (dclist_count(&MXactCache) > MAX_CACHE_ENTRIES)
{
dlist_node *node;
- node = dlist_tail_node(&MXactCache);
- dlist_delete(node);
- MXactCacheMembers--;
+ node = dclist_tail_node(&MXactCache);
+ dclist_delete_from(&MXactCache, node);
entry = dlist_container(mXactCacheEnt, node, node);
debug_elog3(DEBUG2, "CachePut: pruning cached multi %u",
@@ -1699,8 +1697,7 @@ AtEOXact_MultiXact(void)
* a child of TopTransactionContext, we needn't delete it explicitly.
*/
MXactContext = NULL;
- dlist_init(&MXactCache);
- MXactCacheMembers = 0;
+ dclist_init(&MXactCache);
}
/*
@@ -1766,8 +1763,7 @@ PostPrepare_MultiXact(TransactionId xid)
* Discard the local MultiXactId cache like in AtEOXact_MultiXact.
*/
MXactContext = NULL;
- dlist_init(&MXactCache);
- MXactCacheMembers = 0;
+ dclist_init(&MXactCache);
}
/*
diff --git a/src/backend/lib/ilist.c b/src/backend/lib/ilist.c
index 29ef216212..eeb53d3110 100644
--- a/src/backend/lib/ilist.c
+++ b/src/backend/lib/ilist.c
@@ -52,6 +52,21 @@ slist_delete(slist_head *head, slist_node *node)
}
#ifdef ILIST_DEBUG
+/*
+ * dlist_member_check
+ * Validate that 'node' is a member of 'head'
+ */
+void
+dlist_member_check(dlist_head *head, dlist_node *node)
+{
+ for (dlist_node *cur = &head->head; cur != NULL; cur = cur->next)
+ {
+ if (cur == node)
+ return;
+ }
+ elog(ERROR, "double linked list member check failure");
+}
+
/*
* Verify integrity of a doubly linked list
*/
@@ -107,5 +122,4 @@ slist_check(slist_head *head)
for (cur = head->head.next; cur != NULL; cur = cur->next)
;
}
-
#endif /* ILIST_DEBUG */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c29894041e..0377f6d789 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -349,8 +349,6 @@ ReorderBufferAllocate(void)
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
- buffer->catchange_ntxns = 0;
-
buffer->outbuf = NULL;
buffer->outbufsize = 0;
buffer->size = 0;
@@ -368,7 +366,7 @@ ReorderBufferAllocate(void)
dlist_init(&buffer->toplevel_by_lsn);
dlist_init(&buffer->txns_by_base_snapshot_lsn);
- dlist_init(&buffer->catchange_txns);
+ dclist_init(&buffer->catchange_txns);
/*
* Ensure there's no stale data from prior uses of this slot, in case some
@@ -1553,12 +1551,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
*/
dlist_delete(&txn->node);
if (rbtxn_has_catalog_changes(txn))
- {
- dlist_delete(&txn->catchange_node);
- rb->catchange_ntxns--;
-
- Assert(rb->catchange_ntxns >= 0);
- }
+ dclist_delete_from(&rb->catchange_txns, &txn->catchange_node);
/* now remove reference from buffer */
hash_search(rb->by_txn,
@@ -3309,8 +3302,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (!rbtxn_has_catalog_changes(txn))
{
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &txn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &txn->catchange_node);
}
/*
@@ -3323,8 +3315,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (toptxn != NULL && !rbtxn_has_catalog_changes(toptxn))
{
toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
}
}
@@ -3342,19 +3333,17 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
size_t xcnt = 0;
/* Quick return if the list is empty */
- if (rb->catchange_ntxns == 0)
- {
- Assert(dlist_is_empty(&rb->catchange_txns));
+ if (dclist_count(&rb->catchange_txns) == 0)
return NULL;
- }
/* Initialize XID array */
- xids = (TransactionId *) palloc(sizeof(TransactionId) * rb->catchange_ntxns);
- dlist_foreach(iter, &rb->catchange_txns)
+ xids = (TransactionId *) palloc(sizeof(TransactionId) *
+ dclist_count(&rb->catchange_txns));
+ dclist_foreach(iter, &rb->catchange_txns)
{
- ReorderBufferTXN *txn = dlist_container(ReorderBufferTXN,
- catchange_node,
- iter.cur);
+ ReorderBufferTXN *txn = dclist_container(ReorderBufferTXN,
+ catchange_node,
+ iter.cur);
Assert(rbtxn_has_catalog_changes(txn));
@@ -3363,7 +3352,7 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
qsort(xids, xcnt, sizeof(TransactionId), xidComparator);
- Assert(xcnt == rb->catchange_ntxns);
+ Assert(xcnt == dclist_count(&rb->catchange_txns));
return xids;
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index f892d19bfb..5006a5c464 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1688,7 +1688,7 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
/* Get the catalog modifying transactions that are yet not committed */
catchange_xip = ReorderBufferGetCatalogChangesXacts(builder->reorder);
- catchange_xcnt = builder->reorder->catchange_ntxns;
+ catchange_xcnt = dclist_count(&builder->reorder->catchange_txns);
needed_length = sizeof(SnapBuildOnDisk) +
sizeof(TransactionId) * (builder->committed.xcnt + catchange_xcnt);
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index d6f660edf7..5a3aca4aef 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -70,16 +70,13 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
- {
- Assert(dlist_is_empty(&xact_state->pending_drops));
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
- }
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
xl_xact_stats_item *it = &pending->item;
if (isCommit && !pending->is_create)
@@ -101,8 +98,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
not_freed_count++;
}
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
pfree(pending);
}
@@ -144,19 +140,18 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
parent_xact_state = pgstat_get_xact_stack_level(nestDepth - 1);
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
xl_xact_stats_item *it = &pending->item;
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
if (!isCommit && pending->is_create)
{
@@ -175,8 +170,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
* remove the stats object, the surrounding transaction might
* still abort. Pass it on to the parent.
*/
- dlist_push_tail(&parent_xact_state->pending_drops, &pending->node);
- parent_xact_state->pending_drops_count++;
+ dclist_push_tail(&parent_xact_state->pending_drops, &pending->node);
}
else
{
@@ -184,7 +178,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
}
}
- Assert(xact_state->pending_drops_count == 0);
+ Assert(dclist_count(&xact_state->pending_drops) == 0);
if (not_freed_count > 0)
pgstat_request_entry_refs_gc();
}
@@ -250,8 +244,7 @@ pgstat_get_xact_stack_level(int nest_level)
xact_state = (PgStat_SubXactStatus *)
MemoryContextAlloc(TopTransactionContext,
sizeof(PgStat_SubXactStatus));
- dlist_init(&xact_state->pending_drops);
- xact_state->pending_drops_count = 0;
+ dclist_init(&xact_state->pending_drops);
xact_state->nest_level = nest_level;
xact_state->prev = pgStatXactStack;
xact_state->first = NULL;
@@ -291,20 +284,20 @@ pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items)
Assert(!isCommit || xact_state->nest_level == 1);
Assert(!isCommit || xact_state->prev == NULL);
- *items = palloc(xact_state->pending_drops_count
+ *items = palloc(dclist_count(&xact_state->pending_drops)
* sizeof(xl_xact_stats_item));
- dlist_foreach(iter, &xact_state->pending_drops)
+ dclist_foreach(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
if (isCommit && pending->is_create)
continue;
if (!isCommit && !pending->is_create)
continue;
- Assert(nitems < xact_state->pending_drops_count);
+ Assert(nitems < dclist_count(&xact_state->pending_drops));
(*items)[nitems++] = pending->item;
}
@@ -351,8 +344,7 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, Oid objoid, bool
drop->item.dboid = dboid;
drop->item.objoid = objoid;
- dlist_push_tail(&xact_state->pending_drops, &drop->node);
- xact_state->pending_drops_count++;
+ dclist_push_tail(&xact_state->pending_drops, &drop->node);
}
/*
diff --git a/src/backend/utils/adt/ri_triggers.c b/src/backend/utils/adt/ri_triggers.c
index 1d503e7e01..3d24bc7dd6 100644
--- a/src/backend/utils/adt/ri_triggers.c
+++ b/src/backend/utils/adt/ri_triggers.c
@@ -176,8 +176,7 @@ typedef struct RI_CompareHashEntry
static HTAB *ri_constraint_cache = NULL;
static HTAB *ri_query_cache = NULL;
static HTAB *ri_compare_cache = NULL;
-static dlist_head ri_constraint_cache_valid_list;
-static int ri_constraint_cache_valid_count = 0;
+static dclist_head ri_constraint_cache_valid_list;
/*
@@ -2174,8 +2173,7 @@ ri_LoadConstraintInfo(Oid constraintOid)
* For efficient processing of invalidation messages below, we keep a
* doubly-linked list, and a count, of all currently valid entries.
*/
- dlist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
- ri_constraint_cache_valid_count++;
+ dclist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
riinfo->valid = true;
@@ -2233,13 +2231,13 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
* O(N^2) behavior in situations where a session touches many foreign keys
* and also does many ALTER TABLEs, such as a restore from pg_dump.
*/
- if (ri_constraint_cache_valid_count > 1000)
+ if (dclist_count(&ri_constraint_cache_valid_list) > 1000)
hashvalue = 0; /* pretend it's a cache reset */
- dlist_foreach_modify(iter, &ri_constraint_cache_valid_list)
+ dclist_foreach_modify(iter, &ri_constraint_cache_valid_list)
{
- RI_ConstraintInfo *riinfo = dlist_container(RI_ConstraintInfo,
- valid_link, iter.cur);
+ RI_ConstraintInfo *riinfo = dclist_container(RI_ConstraintInfo,
+ valid_link, iter.cur);
/*
* We must invalidate not only entries directly matching the given
@@ -2252,8 +2250,7 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
{
riinfo->valid = false;
/* Remove invalidated entries from the list, too */
- dlist_delete(iter.cur);
- ri_constraint_cache_valid_count--;
+ dclist_delete_from(&ri_constraint_cache_valid_list, iter.cur);
}
}
}
diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index 7ab0888f4f..d8726af6c9 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -10,20 +10,33 @@
* the link fields in the remainder would be wasted space. But usually,
* it saves space to not have separately-allocated list nodes.)
*
+ * The doubly-linked list comes in 2 forms. dlist_head defines a head of a
+ * doubly-linked list of dlist_nodes. Whereas dclist_head defines the head of
+ * a doubly-linked list of dlist_nodes with an additional 'count' field to
+ * keep track of how many items are contained within the given list. For
+ * simplicity dlist_head and dclist_head share the same node type and
+ * iterator types. Each function to manipulate a dlist_head is prefixed by
+ * dlist, whereas functions to manipulate a dclist_head are prefixed with
+ * dclist. dclist_head comes with an additional function (dclist_count) to
+ * return the number of entries in the list. dclists are able to store a
+ * maximum of UINT32 elements. It is up to the caller to ensure no more than
+ * this many items are added to a dclist.
+ *
* None of the functions here allocate any memory; they just manipulate
* externally managed memory. The APIs for singly and doubly linked lists
* are identical as far as capabilities of both allow.
*
* Each list has a list header, which exists even when the list is empty.
* An empty singly-linked list has a NULL pointer in its header.
- * There are two kinds of empty doubly linked lists: those that have been
- * initialized to NULL, and those that have been initialized to circularity.
- * (If a dlist is modified and then all its elements are deleted, it will be
- * in the circular state.) We prefer circular dlists because there are some
- * operations that can be done without branches (and thus faster) on lists
- * that use circular representation. However, it is often convenient to
- * initialize list headers to zeroes rather than setting them up with an
- * explicit initialization function, so we also allow the other case.
+ * For both dlist_head and dclist_head there are two kinds of empty doubly
+ * linked lists: those that have been initialized to NULL, and those that have
+ * been initialized to circularity. (If a dlist is modified and then all its
+ * elements are deleted, it will be in the circular state.) We prefer circular
+ * dlists because there are some operations that can be done without branches
+ * (and thus faster) on lists that use circular representation. However, it
+ * is often convenient to initialize list headers to zeroes rather than
+ * setting them up with an explicit initialization function, so we also allow
+ * the other case.
*
* EXAMPLES
*
@@ -182,6 +195,19 @@ typedef struct dlist_mutable_iter
dlist_node *end; /* last node we'll iterate to */
} dlist_mutable_iter;
+/*
+ * Head of a doubly linked list with a count of the number of items
+ *
+ * This internally makes use of a dlist to implement the actual list and
+ * wrapper functions are used to maintain the count when items are added and
+ * removed from the list.
+ */
+typedef struct dclist_head
+{
+ dlist_head dlist; /* the actual list header */
+ uint32 count; /* the number of items in the list */
+} dclist_head;
+
/*
* Node of a singly linked list.
*
@@ -246,6 +272,7 @@ typedef struct slist_mutable_iter
/* Static initializers */
#define DLIST_STATIC_INIT(name) {{&(name).head, &(name).head}}
+#define DCLIST_STATIC_INIT(name) {{{&(name).dlist.head, &(name).dlist.head}}, 0}
#define SLIST_STATIC_INIT(name) {{NULL}}
@@ -255,6 +282,7 @@ typedef struct slist_mutable_iter
extern void slist_delete(slist_head *head, slist_node *node);
#ifdef ILIST_DEBUG
+extern void dlist_member_check(dlist_head *head, dlist_node *node);
extern void dlist_check(dlist_head *head);
extern void slist_check(slist_head *head);
#else
@@ -264,6 +292,7 @@ extern void slist_check(slist_head *head);
* in which functions the only point of passing the list head pointer is to be
* able to run these checks.
*/
+#define dlist_member_check(head, node) ((void) (head))
#define dlist_check(head) ((void) (head))
#define slist_check(head) ((void) (head))
#endif /* ILIST_DEBUG */
@@ -361,6 +390,17 @@ dlist_delete(dlist_node *node)
node->next->prev = node->prev;
}
+/*
+ * As dlist_delete but performs checks in ILIST_DEBUG to ensure that 'node'
+ * belongs to 'head'.
+ */
+static inline void
+dlist_delete_from(dlist_head *head, dlist_node *node)
+{
+ dlist_member_check(head, node);
+ dlist_delete(node);
+}
+
/*
* Remove and return the first node from a list (there must be one).
*/
@@ -562,6 +602,299 @@ dlist_tail_node(dlist_head *head)
(iter).cur != (iter).end; \
(iter).cur = (iter).cur->prev)
+/* double linked list with count implementation */
+
+/*
+ * Initialize a doubly linked count list.
+ * Previous state will be thrown away without any cleanup.
+ */
+static inline void
+dclist_init(dclist_head *head)
+{
+ dlist_init(&head->dlist);
+ head->count = 0;
+}
+
+/*
+ * dclist_is_empty
+ * Returns true if the list is empty, otherwise false.
+ */
+static inline bool
+dclist_is_empty(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+
+ return (head->count == 0);
+}
+
+/*
+ * dclist_push_head
+ * Insert a node at the beginning of the list.
+ */
+static inline void
+dclist_push_head(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_head(&head->dlist, node);
+ head->count++;
+}
+
+/*
+ * dclist_push_tail
+ * Insert a node at the end of the list.
+ */
+static inline void
+dclist_push_tail(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_tail(&head->dlist, node);
+ head->count++;
+}
+
+/*
+ * dclist_insert_after
+ * Insert a node after another *in the same list*
+ *
+ * Caller must ensure that 'after' is a member of 'head'.
+ */
+static inline void
+dclist_insert_after(dclist_head *head, dlist_node *after, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, after);
+ Assert(head->count > 0);
+
+ dlist_insert_after(after, node);
+ head->count++;
+}
+
+/*
+ * dclist_insert_before
+ * Insert a node before another *in the same list*
+ *
+ * Caller must ensure that 'before' is a member of 'head'.
+ */
+static inline void
+dclist_insert_before(dclist_head *head, dlist_node *before, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, before);
+ Assert(head->count > 0);
+
+ dlist_insert_before(before, node);
+ head->count++;
+}
+
+/*
+ * dclist_delete_from
+ * Deletes 'node' from 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_delete_from(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+ dlist_delete_from(&head->dlist, node);
+ head->count--;
+}
+
+/*
+ * dclist_pop_head_node
+ * Remove and return the first node from a list (there must be one).
+ */
+static inline dlist_node *
+dclist_pop_head_node(dclist_head *head)
+{
+ dlist_node *node;
+
+ Assert(head->count > 0);
+
+ node = dlist_pop_head_node(&head->dlist);
+ head->count--;
+
+ return node;
+}
+
+/*
+ * dclist_move_head
+ * Move 'node' from its current position in the list to the head position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_head(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ dlist_move_head(&head->dlist, node);
+}
+
+/*
+ * dclist_move_tail
+ * Move 'node' from its current position in the list to the tail position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_tail(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ dlist_move_tail(&head->dlist, node);
+}
+
+/*
+ * dclist_has_next
+ * Check whether 'node' has a following node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_next(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ return dlist_has_next(&head->dlist, node);
+}
+
+/*
+ * dclist_has_prev
+ * Check whether 'node' has a preceding node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_prev(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+
+ Assert(head->count > 0);
+
+ return dlist_has_prev(&head->dlist, node);
+}
+
+/*
+ * dclist_next_node
+ * Return the next node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_next_node(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+
+ return dlist_next_node(&head->dlist, node);
+}
+
+/*
+ * dclist_prev_node
+ * Return the prev node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_prev_node(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+
+ return dlist_prev_node(&head->dlist, node);
+}
+
+/* internal support function to get address of head element's struct */
+static inline void *
+dclist_head_element_off(dclist_head *head, size_t off)
+{
+ Assert(!dclist_is_empty(head));
+ return (char *) head->dlist.head.next - off;
+}
+
+/*
+ * dclist_head_node
+ * Return the first node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_head_node(dclist_head *head)
+{
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_head_element_off(&head->dlist, 0);
+}
+
+/* internal support function to get address of tail element's struct */
+static inline void *
+dclist_tail_element_off(dclist_head *head, size_t off)
+{
+ Assert(!dclist_is_empty(head));
+ return (char *) head->dlist.head.prev - off;
+}
+
+/*
+ * Return the last node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_tail_node(dclist_head *head)
+{
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_tail_element_off(&head->dlist, 0);
+}
+
+/*
+ * dclist_count
+ * Returns the stored number of entries in 'head'
+ */
+static inline uint32
+dclist_count(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+
+ return head->count;
+}
+
+/*
+ * Return the containing struct of 'type' where 'membername' is the dlist_node
+ * pointed at by 'ptr'.
+ *
+ * This is used to convert a dlist_node * back to its containing struct.
+ *
+ * Note: This is effectively just the same as dlist_container, so reuse that.
+ */
+#define dclist_container(type, membername, ptr) \
+ dlist_container(type, membername, ptr)
+
+ /*
+ * Return the address of the first element in the list.
+ *
+ * The list must not be empty.
+ */
+#define dclist_head_element(type, membername, lhead) \
+ (AssertVariableIsOfTypeMacro(((type *) NULL)->membername, dlist_node), \
+ (type *) dclist_head_element_off(lhead, offsetof(type, membername)))
+
+ /*
+ * Return the address of the last element in the list.
+ *
+ * The list must not be empty.
+ */
+#define dclist_tail_element(type, membername, lhead) \
+ (AssertVariableIsOfTypeMacro(((type *) NULL)->membername, dlist_node), \
+ ((type *) dclist_tail_element_off(lhead, offsetof(type, membername))))
+
+
+/* Iterators for dclists */
+#define dclist_foreach(iter, lhead) \
+ dlist_foreach(iter, &((lhead)->dlist))
+
+#define dclist_foreach_modify(iter, lhead) \
+ dlist_foreach_modify(iter, &((lhead)->dlist))
+
+#define dclist_reverse_foreach(iter, lhead) \
+ dlist_reverse_foreach(iter, &((lhead)->dlist))
/* singly linked list implementation */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 02b59a1931..b23d8cc4f9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -534,8 +534,7 @@ struct ReorderBuffer
/*
* Transactions and subtransactions that have modified system catalogs.
*/
- dlist_head catchange_txns;
- int catchange_ntxns;
+ dclist_head catchange_txns;
/*
* one-entry sized cache for by_txn. Very frequently the same txn gets
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index 627c1389e4..9f32a6f6d3 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -162,8 +162,7 @@ typedef struct PgStat_SubXactStatus
* if the transaction commits/aborts. To handle replicas and crashes,
* stats drops are included in commit / abort records.
*/
- dlist_head pending_drops;
- int pending_drops_count;
+ dclist_head pending_drops;
/*
* Tuple insertion/deletion counts for an open transaction can't be
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2f02cc8f42..c15f990cbb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3194,6 +3194,7 @@ datapagemap_t
dateKEY
datetkn
dce_uuid_t
+dclist_head
decimal
deparse_columns
deparse_context
On Fri, Oct 28, 2022 at 11:01 AM David Rowley <dgrowleyml@gmail.com> wrote:
3. Just thinking if we need to move dlist_is_memberof() check from
dclist_* functions to dlist_* functions, because they also need such
insurance against callers passing spurious nodes.I think the affected functions there would be; dlist_move_head(),
dlist_move_tail(), dlist_has_next(), dlist_has_prev(),
dlist_next_node() and dlist_prev_node(). I believe if we did that then
it's effectively an API change. The comments only claim that it's
undefined if node is not a member of the list. It does not say 'node'
*must* be part of the list. Now, perhaps doing this would just make
it more likely that we'd find bugs in our code and extension authors
would find bugs in their code, but it does move the bar.
dlist_move_head and dlist_move_tail look like they'd work perfectly
well to remove an item from 1 list and put it on the head or tail of
some completely different list. Should we really be changing that in a
patch that is meant to just add the dclist type?
Hm. Let's not touch that here.
Thanks for the patch. Few more comments on v2:
1. I guess we need to cast the 'node' parameter too, something like
below? I'm reading the comment there talking about compilers
complaning about the unused function arguments.
dlist_member_check(head, node) ((void) (head); (void) (node);)
2.
+ * maximum of UINT32 elements. It is up to the caller to ensure no more than
+ * this many items are added to a dclist.
Can we put max limit, at least in assert, something like below, on
'count' instead of saying above? I'm not sure if there's someone
storing 4 billion items, but it will be a good-to-have safety from the
data structure perspective if others think it's not an overkill.
Assert(head->count > 0 && head->count <= PG_UINT32_MAX);
3. I guess, we can split up the patches for the ease of review, 0001
introducing dclist_* data structure and 0002 using it. It's not
mandatory though. The two patches can go separately if needed.
4.
+/*
+ * As dlist_delete but performs checks in ILIST_DEBUG to ensure that 'node'
+ * belongs to 'head'.
I think 'Same as dlist_delete' instead of just 'As dlist_delete'
5.
+ * Caution: 'node' must be a member of 'head'.
+ * Caller must ensure that 'before' is a member of 'head'.
Can we have the same comments around something like below?
+ * Caller must ensure that 'node' must be a member of 'head'.
+ * Caller must ensure that 'before' is a member of 'head'.
or
+ * Caution: 'node' must be a member of 'head'.
+ * Caution: 'before' must be a member of 'head'.
or
* Caution: unreliable if 'node' is not in the list.
* Caution: unreliable if 'before' is not in the list.
6.
+dclist_has_prev(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+
+ Assert(head->count > 0);
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_head_element_off(&head->dlist, 0);
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_tail_element_off(&head->dlist, 0);
+ Assert(!dclist_is_empty(head));
+ return (char *) head->dlist.head.next - off;
+ dlist_member_check(&head->dlist, node);
+
+ Assert(head->count > 0);
+
+ return dlist_has_prev(&head->dlist, node);
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ return dlist_has_next(&head->dlist, node);
Remove extra lines in between and have them uniformly across,
something like below?
dlist_member_check();
Assert();
return XXX;
8. Wondering if we need dlist_delete_from() at all. Can't we just add
dlist_member_check() dclist_delete_from() and call dlist_delete()
directly?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Thank you for having another look at this
On Sat, 29 Oct 2022 at 18:32, Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
1. I guess we need to cast the 'node' parameter too, something like
below? I'm reading the comment there talking about compilers
complaning about the unused function arguments.
dlist_member_check(head, node) ((void) (head); (void) (node);)
I looked at dlist_check() and I didn't quite manage to figure out why
the cast is needed. As far as I can see, there are no calls where we
only pass dlist_head solely for the dlist_check(). For
dlist_member_check(), dlist_delete_from() does not use the 'head'
parameter for anything apart from dlist_member_check(), so I believe
the cast is required for 'head'. I think I'd rather only add the cast
for 'node' unless we really require it. Cargo-culting it in there just
because that's what the other macros do does not seem like a good idea
to me.
Can we put max limit, at least in assert, something like below, on
'count' instead of saying above? I'm not sure if there's someone
storing 4 billion items, but it will be a good-to-have safety from the
data structure perspective if others think it's not an overkill.
Assert(head->count > 0 && head->count <= PG_UINT32_MAX);
'count' is a uint32. It's always going to be <= PG_UINT32_MAX.
My original thoughts were that it seems unlikely we'd ever give an
assert build a workload that would ever have 2^32 dlist_nodes in
dclist. Having said that, perhaps it would do no harm to add some
overflow checks to 'count'. I've gone and added some
Assert(head->count > 0) after we do count++.
+ * As dlist_delete but performs checks in ILIST_DEBUG to ensure that 'node' + * belongs to 'head'. I think 'Same as dlist_delete' instead of just 'As dlist_delete'
I don't really see what's wrong with this. We use "As above" when we
mean "Same as above" in many locations. Anyway, I don't feel strongly
about not adding the word, so I've adjusted the wording in that
comment which includes adding the word "Same" at the start.
5. + * Caution: 'node' must be a member of 'head'. + * Caller must ensure that 'before' is a member of 'head'. Can we have the same comments around something like below?
I've adjusted dclist_insert_after() and dclist_insert_before(). Each
dclist function that uses dlist_member_check() now has the same text.
8. Wondering if we need dlist_delete_from() at all. Can't we just add
dlist_member_check() dclist_delete_from() and call dlist_delete()
directly?
Certainly, but I made it that way on purpose. I wanted dclist to have
a superset of the functions that dlist has. I just see no reason why
dlist shouldn't have dlist_delete_from() when dclist has it.
I've attached the v3 version of the patch which includes some
additional polishing work.
David
Attachments:
v3_add_doubly_linked_count_lists.patchtext/plain; charset=US-ASCII; name=v3_add_doubly_linked_count_lists.patchDownload
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index b01b39b008..a34e9b352d 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -196,8 +196,7 @@ typedef struct RewriteMappingFile
TransactionId xid; /* xid that might need to see the row */
int vfd; /* fd of mappings file */
off_t off; /* how far have we written yet */
- uint32 num_mappings; /* number of in-memory mappings */
- dlist_head mappings; /* list of in-memory mappings */
+ dclist_head mappings; /* list of in-memory mappings */
char path[MAXPGPATH]; /* path, for error messages */
} RewriteMappingFile;
@@ -864,9 +863,10 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
Oid dboid;
uint32 len;
int written;
+ uint32 num_mappings = dclist_count(&src->mappings);
/* this file hasn't got any new mappings */
- if (src->num_mappings == 0)
+ if (num_mappings == 0)
continue;
if (state->rs_old_rel->rd_rel->relisshared)
@@ -874,7 +874,7 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
else
dboid = MyDatabaseId;
- xlrec.num_mappings = src->num_mappings;
+ xlrec.num_mappings = num_mappings;
xlrec.mapped_rel = RelationGetRelid(state->rs_old_rel);
xlrec.mapped_xid = src->xid;
xlrec.mapped_db = dboid;
@@ -882,31 +882,30 @@ logical_heap_rewrite_flush_mappings(RewriteState state)
xlrec.start_lsn = state->rs_begin_lsn;
/* write all mappings consecutively */
- len = src->num_mappings * sizeof(LogicalRewriteMappingData);
+ len = num_mappings * sizeof(LogicalRewriteMappingData);
waldata_start = waldata = palloc(len);
/*
* collect data we need to write out, but don't modify ondisk data yet
*/
- dlist_foreach_modify(iter, &src->mappings)
+ dclist_foreach_modify(iter, &src->mappings)
{
RewriteMappingDataEntry *pmap;
- pmap = dlist_container(RewriteMappingDataEntry, node, iter.cur);
+ pmap = dclist_container(RewriteMappingDataEntry, node, iter.cur);
memcpy(waldata, &pmap->map, sizeof(pmap->map));
waldata += sizeof(pmap->map);
/* remove from the list and free */
- dlist_delete(&pmap->node);
+ dclist_delete_from(&src->mappings, &pmap->node);
pfree(pmap);
/* update bookkeeping */
state->rs_num_rewrite_mappings--;
- src->num_mappings--;
}
- Assert(src->num_mappings == 0);
+ Assert(dclist_count(&src->mappings) == 0);
Assert(waldata == waldata_start + len);
/*
@@ -1002,8 +1001,7 @@ logical_rewrite_log_mapping(RewriteState state, TransactionId xid,
LSN_FORMAT_ARGS(state->rs_begin_lsn),
xid, GetCurrentTransactionId());
- dlist_init(&src->mappings);
- src->num_mappings = 0;
+ dclist_init(&src->mappings);
src->off = 0;
memcpy(src->path, path, sizeof(path));
src->vfd = PathNameOpenFile(path,
@@ -1017,8 +1015,7 @@ logical_rewrite_log_mapping(RewriteState state, TransactionId xid,
pmap = MemoryContextAlloc(state->rs_cxt,
sizeof(RewriteMappingDataEntry));
memcpy(&pmap->map, map, sizeof(LogicalRewriteMappingData));
- dlist_push_tail(&src->mappings, &pmap->node);
- src->num_mappings++;
+ dclist_push_tail(&src->mappings, &pmap->node);
state->rs_num_rewrite_mappings++;
/*
diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c
index 5eff87665b..09c9409c31 100644
--- a/src/backend/access/transam/multixact.c
+++ b/src/backend/access/transam/multixact.c
@@ -319,8 +319,7 @@ typedef struct mXactCacheEnt
} mXactCacheEnt;
#define MAX_CACHE_ENTRIES 256
-static dlist_head MXactCache = DLIST_STATIC_INIT(MXactCache);
-static int MXactCacheMembers = 0;
+static dclist_head MXactCache = DCLIST_STATIC_INIT(MXactCache);
static MemoryContext MXactContext = NULL;
#ifdef MULTIXACT_DEBUG
@@ -1504,9 +1503,9 @@ mXactCacheGetBySet(int nmembers, MultiXactMember *members)
/* sort the array so comparison is easy */
qsort(members, nmembers, sizeof(MultiXactMember), mxactMemberComparator);
- dlist_foreach(iter, &MXactCache)
+ dclist_foreach(iter, &MXactCache)
{
- mXactCacheEnt *entry = dlist_container(mXactCacheEnt, node, iter.cur);
+ mXactCacheEnt *entry = dclist_container(mXactCacheEnt, node, iter.cur);
if (entry->nmembers != nmembers)
continue;
@@ -1518,7 +1517,7 @@ mXactCacheGetBySet(int nmembers, MultiXactMember *members)
if (memcmp(members, entry->members, nmembers * sizeof(MultiXactMember)) == 0)
{
debug_elog3(DEBUG2, "CacheGet: found %u", entry->multi);
- dlist_move_head(&MXactCache, iter.cur);
+ dclist_move_head(&MXactCache, iter.cur);
return entry->multi;
}
}
@@ -1542,9 +1541,9 @@ mXactCacheGetById(MultiXactId multi, MultiXactMember **members)
debug_elog3(DEBUG2, "CacheGet: looking for %u", multi);
- dlist_foreach(iter, &MXactCache)
+ dclist_foreach(iter, &MXactCache)
{
- mXactCacheEnt *entry = dlist_container(mXactCacheEnt, node, iter.cur);
+ mXactCacheEnt *entry = dclist_container(mXactCacheEnt, node, iter.cur);
if (entry->multi == multi)
{
@@ -1566,7 +1565,7 @@ mXactCacheGetById(MultiXactId multi, MultiXactMember **members)
* This is acceptable only because we exit the iteration
* immediately afterwards.
*/
- dlist_move_head(&MXactCache, iter.cur);
+ dclist_move_head(&MXactCache, iter.cur);
*members = ptr;
return entry->nmembers;
@@ -1610,16 +1609,15 @@ mXactCachePut(MultiXactId multi, int nmembers, MultiXactMember *members)
/* mXactCacheGetBySet assumes the entries are sorted, so sort them */
qsort(entry->members, nmembers, sizeof(MultiXactMember), mxactMemberComparator);
- dlist_push_head(&MXactCache, &entry->node);
- if (MXactCacheMembers++ >= MAX_CACHE_ENTRIES)
+ dclist_push_head(&MXactCache, &entry->node);
+ if (dclist_count(&MXactCache) > MAX_CACHE_ENTRIES)
{
dlist_node *node;
- node = dlist_tail_node(&MXactCache);
- dlist_delete(node);
- MXactCacheMembers--;
+ node = dclist_tail_node(&MXactCache);
+ dclist_delete_from(&MXactCache, node);
- entry = dlist_container(mXactCacheEnt, node, node);
+ entry = dclist_container(mXactCacheEnt, node, node);
debug_elog3(DEBUG2, "CachePut: pruning cached multi %u",
entry->multi);
@@ -1699,8 +1697,7 @@ AtEOXact_MultiXact(void)
* a child of TopTransactionContext, we needn't delete it explicitly.
*/
MXactContext = NULL;
- dlist_init(&MXactCache);
- MXactCacheMembers = 0;
+ dclist_init(&MXactCache);
}
/*
@@ -1766,8 +1763,7 @@ PostPrepare_MultiXact(TransactionId xid)
* Discard the local MultiXactId cache like in AtEOXact_MultiXact.
*/
MXactContext = NULL;
- dlist_init(&MXactCache);
- MXactCacheMembers = 0;
+ dclist_init(&MXactCache);
}
/*
diff --git a/src/backend/lib/ilist.c b/src/backend/lib/ilist.c
index 29ef216212..e8ea981176 100644
--- a/src/backend/lib/ilist.c
+++ b/src/backend/lib/ilist.c
@@ -52,6 +52,23 @@ slist_delete(slist_head *head, slist_node *node)
}
#ifdef ILIST_DEBUG
+/*
+ * dlist_member_check
+ * Validate that 'node' is a member of 'head'
+ */
+void
+dlist_member_check(dlist_head *head, dlist_node *node)
+{
+ dlist_iter iter;
+
+ dlist_foreach(iter, head)
+ {
+ if (iter.cur == node)
+ return;
+ }
+ elog(ERROR, "double linked list member check failure");
+}
+
/*
* Verify integrity of a doubly linked list
*/
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 20c9cd139a..22a15a482a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -349,8 +349,6 @@ ReorderBufferAllocate(void)
buffer->by_txn_last_xid = InvalidTransactionId;
buffer->by_txn_last_txn = NULL;
- buffer->catchange_ntxns = 0;
-
buffer->outbuf = NULL;
buffer->outbufsize = 0;
buffer->size = 0;
@@ -368,7 +366,7 @@ ReorderBufferAllocate(void)
dlist_init(&buffer->toplevel_by_lsn);
dlist_init(&buffer->txns_by_base_snapshot_lsn);
- dlist_init(&buffer->catchange_txns);
+ dclist_init(&buffer->catchange_txns);
/*
* Ensure there's no stale data from prior uses of this slot, in case some
@@ -1553,12 +1551,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
*/
dlist_delete(&txn->node);
if (rbtxn_has_catalog_changes(txn))
- {
- dlist_delete(&txn->catchange_node);
- rb->catchange_ntxns--;
-
- Assert(rb->catchange_ntxns >= 0);
- }
+ dclist_delete_from(&rb->catchange_txns, &txn->catchange_node);
/* now remove reference from buffer */
hash_search(rb->by_txn,
@@ -3309,8 +3302,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (!rbtxn_has_catalog_changes(txn))
{
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &txn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &txn->catchange_node);
}
/*
@@ -3323,8 +3315,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
if (toptxn != NULL && !rbtxn_has_catalog_changes(toptxn))
{
toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- dlist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
- rb->catchange_ntxns++;
+ dclist_push_tail(&rb->catchange_txns, &toptxn->catchange_node);
}
}
@@ -3342,19 +3333,17 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
size_t xcnt = 0;
/* Quick return if the list is empty */
- if (rb->catchange_ntxns == 0)
- {
- Assert(dlist_is_empty(&rb->catchange_txns));
+ if (dclist_count(&rb->catchange_txns) == 0)
return NULL;
- }
/* Initialize XID array */
- xids = (TransactionId *) palloc(sizeof(TransactionId) * rb->catchange_ntxns);
- dlist_foreach(iter, &rb->catchange_txns)
+ xids = (TransactionId *) palloc(sizeof(TransactionId) *
+ dclist_count(&rb->catchange_txns));
+ dclist_foreach(iter, &rb->catchange_txns)
{
- ReorderBufferTXN *txn = dlist_container(ReorderBufferTXN,
- catchange_node,
- iter.cur);
+ ReorderBufferTXN *txn = dclist_container(ReorderBufferTXN,
+ catchange_node,
+ iter.cur);
Assert(rbtxn_has_catalog_changes(txn));
@@ -3363,7 +3352,7 @@ ReorderBufferGetCatalogChangesXacts(ReorderBuffer *rb)
qsort(xids, xcnt, sizeof(TransactionId), xidComparator);
- Assert(xcnt == rb->catchange_ntxns);
+ Assert(xcnt == dclist_count(&rb->catchange_txns));
return xids;
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index f892d19bfb..5006a5c464 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -1688,7 +1688,7 @@ SnapBuildSerialize(SnapBuild *builder, XLogRecPtr lsn)
/* Get the catalog modifying transactions that are yet not committed */
catchange_xip = ReorderBufferGetCatalogChangesXacts(builder->reorder);
- catchange_xcnt = builder->reorder->catchange_ntxns;
+ catchange_xcnt = dclist_count(&builder->reorder->catchange_txns);
needed_length = sizeof(SnapBuildOnDisk) +
sizeof(TransactionId) * (builder->committed.xcnt + catchange_xcnt);
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index d6f660edf7..5a3aca4aef 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -70,16 +70,13 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
- {
- Assert(dlist_is_empty(&xact_state->pending_drops));
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
- }
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
xl_xact_stats_item *it = &pending->item;
if (isCommit && !pending->is_create)
@@ -101,8 +98,7 @@ AtEOXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state, bool isCommit)
not_freed_count++;
}
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
pfree(pending);
}
@@ -144,19 +140,18 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
dlist_mutable_iter iter;
int not_freed_count = 0;
- if (xact_state->pending_drops_count == 0)
+ if (dclist_count(&xact_state->pending_drops) == 0)
return;
parent_xact_state = pgstat_get_xact_stack_level(nestDepth - 1);
- dlist_foreach_modify(iter, &xact_state->pending_drops)
+ dclist_foreach_modify(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
xl_xact_stats_item *it = &pending->item;
- dlist_delete(&pending->node);
- xact_state->pending_drops_count--;
+ dclist_delete_from(&xact_state->pending_drops, &pending->node);
if (!isCommit && pending->is_create)
{
@@ -175,8 +170,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
* remove the stats object, the surrounding transaction might
* still abort. Pass it on to the parent.
*/
- dlist_push_tail(&parent_xact_state->pending_drops, &pending->node);
- parent_xact_state->pending_drops_count++;
+ dclist_push_tail(&parent_xact_state->pending_drops, &pending->node);
}
else
{
@@ -184,7 +178,7 @@ AtEOSubXact_PgStat_DroppedStats(PgStat_SubXactStatus *xact_state,
}
}
- Assert(xact_state->pending_drops_count == 0);
+ Assert(dclist_count(&xact_state->pending_drops) == 0);
if (not_freed_count > 0)
pgstat_request_entry_refs_gc();
}
@@ -250,8 +244,7 @@ pgstat_get_xact_stack_level(int nest_level)
xact_state = (PgStat_SubXactStatus *)
MemoryContextAlloc(TopTransactionContext,
sizeof(PgStat_SubXactStatus));
- dlist_init(&xact_state->pending_drops);
- xact_state->pending_drops_count = 0;
+ dclist_init(&xact_state->pending_drops);
xact_state->nest_level = nest_level;
xact_state->prev = pgStatXactStack;
xact_state->first = NULL;
@@ -291,20 +284,20 @@ pgstat_get_transactional_drops(bool isCommit, xl_xact_stats_item **items)
Assert(!isCommit || xact_state->nest_level == 1);
Assert(!isCommit || xact_state->prev == NULL);
- *items = palloc(xact_state->pending_drops_count
+ *items = palloc(dclist_count(&xact_state->pending_drops)
* sizeof(xl_xact_stats_item));
- dlist_foreach(iter, &xact_state->pending_drops)
+ dclist_foreach(iter, &xact_state->pending_drops)
{
PgStat_PendingDroppedStatsItem *pending =
- dlist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
+ dclist_container(PgStat_PendingDroppedStatsItem, node, iter.cur);
if (isCommit && pending->is_create)
continue;
if (!isCommit && !pending->is_create)
continue;
- Assert(nitems < xact_state->pending_drops_count);
+ Assert(nitems < dclist_count(&xact_state->pending_drops));
(*items)[nitems++] = pending->item;
}
@@ -351,8 +344,7 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, Oid objoid, bool
drop->item.dboid = dboid;
drop->item.objoid = objoid;
- dlist_push_tail(&xact_state->pending_drops, &drop->node);
- xact_state->pending_drops_count++;
+ dclist_push_tail(&xact_state->pending_drops, &drop->node);
}
/*
diff --git a/src/backend/utils/adt/ri_triggers.c b/src/backend/utils/adt/ri_triggers.c
index 1d503e7e01..61c2eecaca 100644
--- a/src/backend/utils/adt/ri_triggers.c
+++ b/src/backend/utils/adt/ri_triggers.c
@@ -176,8 +176,7 @@ typedef struct RI_CompareHashEntry
static HTAB *ri_constraint_cache = NULL;
static HTAB *ri_query_cache = NULL;
static HTAB *ri_compare_cache = NULL;
-static dlist_head ri_constraint_cache_valid_list;
-static int ri_constraint_cache_valid_count = 0;
+static dclist_head ri_constraint_cache_valid_list;
/*
@@ -2172,10 +2171,9 @@ ri_LoadConstraintInfo(Oid constraintOid)
/*
* For efficient processing of invalidation messages below, we keep a
- * doubly-linked list, and a count, of all currently valid entries.
+ * doubly-linked count list of all currently valid entries.
*/
- dlist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
- ri_constraint_cache_valid_count++;
+ dclist_push_tail(&ri_constraint_cache_valid_list, &riinfo->valid_link);
riinfo->valid = true;
@@ -2233,13 +2231,13 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
* O(N^2) behavior in situations where a session touches many foreign keys
* and also does many ALTER TABLEs, such as a restore from pg_dump.
*/
- if (ri_constraint_cache_valid_count > 1000)
+ if (dclist_count(&ri_constraint_cache_valid_list) > 1000)
hashvalue = 0; /* pretend it's a cache reset */
- dlist_foreach_modify(iter, &ri_constraint_cache_valid_list)
+ dclist_foreach_modify(iter, &ri_constraint_cache_valid_list)
{
- RI_ConstraintInfo *riinfo = dlist_container(RI_ConstraintInfo,
- valid_link, iter.cur);
+ RI_ConstraintInfo *riinfo = dclist_container(RI_ConstraintInfo,
+ valid_link, iter.cur);
/*
* We must invalidate not only entries directly matching the given
@@ -2252,8 +2250,7 @@ InvalidateConstraintCacheCallBack(Datum arg, int cacheid, uint32 hashvalue)
{
riinfo->valid = false;
/* Remove invalidated entries from the list, too */
- dlist_delete(iter.cur);
- ri_constraint_cache_valid_count--;
+ dclist_delete_from(&ri_constraint_cache_valid_list, iter.cur);
}
}
}
diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index 7ab0888f4f..4ac0afe5e7 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -10,20 +10,36 @@
* the link fields in the remainder would be wasted space. But usually,
* it saves space to not have separately-allocated list nodes.)
*
+ * The doubly-linked list comes in 2 forms. dlist_head defines a head of a
+ * doubly-linked list of dlist_nodes. Whereas dclist_head defines the head of
+ * a doubly-linked list of dlist_nodes with an additional 'count' field to
+ * keep track of how many items are contained within the given list. For
+ * simplicity dlist_head and dclist_head share the same node and iterator
+ * types. The functions to manipulate a dlist_head always have a name
+ * starting with "dlist", whereas functions to manipulate a dclist_head have a
+ * name starting with "dclist". dclist_head comes with an additional function
+ * (dclist_count) to return the number of entries in the list. dclists are
+ * able to store a maximum of PG_UINT32_MAX elements. It is up to the caller
+ * to ensure no more than this many items are added to a dclist.
+ *
* None of the functions here allocate any memory; they just manipulate
- * externally managed memory. The APIs for singly and doubly linked lists
- * are identical as far as capabilities of both allow.
+ * externally managed memory. With the exception doubly-linked count lists
+ * providing the ability to obtain the number of items in the list, the APIs
+ * for singly and both doubly linked lists are identical as far as
+ * capabilities of both allow.
*
* Each list has a list header, which exists even when the list is empty.
* An empty singly-linked list has a NULL pointer in its header.
- * There are two kinds of empty doubly linked lists: those that have been
- * initialized to NULL, and those that have been initialized to circularity.
+ *
+ * For both doubly-linked list types, there are two valid ways to represent an
+ * empty list. The head's 'next' pointer can either be NULL or the head's
+ * 'next' and 'prev' links can both point back to the list head (circular).
* (If a dlist is modified and then all its elements are deleted, it will be
- * in the circular state.) We prefer circular dlists because there are some
+ * in the circular state.). We prefer circular dlists because there are some
* operations that can be done without branches (and thus faster) on lists
* that use circular representation. However, it is often convenient to
* initialize list headers to zeroes rather than setting them up with an
- * explicit initialization function, so we also allow the other case.
+ * explicit initialization function, so we also allow the NULL initalization.
*
* EXAMPLES
*
@@ -146,15 +162,17 @@ typedef struct dlist_head
/*
- * Doubly linked list iterator.
+ * Doubly linked list iterator type for dlist_head and and dclist_head types.
+ *
+ * Used as state in dlist_foreach() and dlist_reverse_foreach() (and the
+ * dclist variant thereof).
*
- * Used as state in dlist_foreach() and dlist_reverse_foreach(). To get the
- * current element of the iteration use the 'cur' member.
+ * To get the current element of the iteration use the 'cur' member.
*
* Iterations using this are *not* allowed to change the list while iterating!
*
* NB: We use an extra "end" field here to avoid multiple evaluations of
- * arguments in the dlist_foreach() macro.
+ * arguments in the dlist_foreach() and dclist_foreach() macros.
*/
typedef struct dlist_iter
{
@@ -163,10 +181,12 @@ typedef struct dlist_iter
} dlist_iter;
/*
- * Doubly linked list iterator allowing some modifications while iterating.
+ * Doubly linked list iterator for both dlist_head and dclist_head types.
+ * This iterator type allows some modifications while iterating.
*
- * Used as state in dlist_foreach_modify(). To get the current element of the
- * iteration use the 'cur' member.
+ * Used as state in dlist_foreach_modify() and dclist_foreach_modify().
+ *
+ * To get the current element of the iteration use the 'cur' member.
*
* Iterations using this are only allowed to change the list at the current
* point of iteration. It is fine to delete the current node, but it is *not*
@@ -182,6 +202,19 @@ typedef struct dlist_mutable_iter
dlist_node *end; /* last node we'll iterate to */
} dlist_mutable_iter;
+/*
+ * Head of a doubly linked list with a count of the number of items
+ *
+ * This internally makes use of a dlist to implement the actual list and
+ * wrapper functions are used to maintain the count when items are added and
+ * removed from the list.
+ */
+typedef struct dclist_head
+{
+ dlist_head dlist; /* the actual list header */
+ uint32 count; /* the number of items in the list */
+} dclist_head;
+
/*
* Node of a singly linked list.
*
@@ -246,6 +279,7 @@ typedef struct slist_mutable_iter
/* Static initializers */
#define DLIST_STATIC_INIT(name) {{&(name).head, &(name).head}}
+#define DCLIST_STATIC_INIT(name) {{{&(name).dlist.head, &(name).dlist.head}}, 0}
#define SLIST_STATIC_INIT(name) {{NULL}}
@@ -255,6 +289,7 @@ typedef struct slist_mutable_iter
extern void slist_delete(slist_head *head, slist_node *node);
#ifdef ILIST_DEBUG
+extern void dlist_member_check(dlist_head *head, dlist_node *node);
extern void dlist_check(dlist_head *head);
extern void slist_check(slist_head *head);
#else
@@ -264,6 +299,7 @@ extern void slist_check(slist_head *head);
* in which functions the only point of passing the list head pointer is to be
* able to run these checks.
*/
+#define dlist_member_check(head, node) ((void) (head))
#define dlist_check(head) ((void) (head))
#define slist_check(head) ((void) (head))
#endif /* ILIST_DEBUG */
@@ -361,6 +397,17 @@ dlist_delete(dlist_node *node)
node->next->prev = node->prev;
}
+/*
+ * Same as dlist_delete, but performs checks in ILIST_DEBUG builds to ensure
+ * that 'node' belongs to 'head'.
+ */
+static inline void
+dlist_delete_from(dlist_head *head, dlist_node *node)
+{
+ dlist_member_check(head, node);
+ dlist_delete(node);
+}
+
/*
* Remove and return the first node from a list (there must be one).
*/
@@ -562,6 +609,309 @@ dlist_tail_node(dlist_head *head)
(iter).cur != (iter).end; \
(iter).cur = (iter).cur->prev)
+/* doubly-linked count list implementation */
+
+/*
+ * dclist_init
+ * Initialize a doubly linked count list.
+ *
+ * Previous state will be thrown away without any cleanup.
+ */
+static inline void
+dclist_init(dclist_head *head)
+{
+ dlist_init(&head->dlist);
+ head->count = 0;
+}
+
+/*
+ * dclist_is_empty
+ * Returns true if the list is empty, otherwise false.
+ */
+static inline bool
+dclist_is_empty(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+ return (head->count == 0);
+}
+
+/*
+ * dclist_push_head
+ * Insert a node at the beginning of the list.
+ */
+static inline void
+dclist_push_head(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_head(&head->dlist, node);
+ head->count++;
+
+ Assert(head->count > 0); /* count overflow check */
+}
+
+/*
+ * dclist_push_tail
+ * Insert a node at the end of the list.
+ */
+static inline void
+dclist_push_tail(dclist_head *head, dlist_node *node)
+{
+ if (head->dlist.head.next == NULL) /* convert NULL header to circular */
+ dclist_init(head);
+
+ dlist_push_tail(&head->dlist, node);
+ head->count++;
+
+ Assert(head->count > 0); /* count overflow check */
+}
+
+/*
+ * dclist_insert_after
+ * Insert a node after another *in the same list*
+ *
+ * Caution: 'after' must be a member of 'head'.
+ */
+static inline void
+dclist_insert_after(dclist_head *head, dlist_node *after, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, after);
+ Assert(head->count > 0); /* must be at least 1 already */
+
+ dlist_insert_after(after, node);
+ head->count++;
+
+ Assert(head->count > 0); /* count overflow check */
+}
+
+/*
+ * dclist_insert_before
+ * Insert a node before another *in the same list*
+ *
+ * Caution: 'before' must be a member of 'head'.
+ */
+static inline void
+dclist_insert_before(dclist_head *head, dlist_node *before, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, before);
+ Assert(head->count > 0); /* must be at least 1 already */
+
+ dlist_insert_before(before, node);
+ head->count++;
+
+ Assert(head->count > 0); /* count overflow check */
+}
+
+/*
+ * dclist_delete_from
+ * Deletes 'node' from 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_delete_from(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+
+ dlist_delete_from(&head->dlist, node);
+ head->count--;
+}
+
+/*
+ * dclist_pop_head_node
+ * Remove and return the first node from a list (there must be one).
+ */
+static inline dlist_node *
+dclist_pop_head_node(dclist_head *head)
+{
+ dlist_node *node;
+
+ Assert(head->count > 0);
+
+ node = dlist_pop_head_node(&head->dlist);
+ head->count--;
+ return node;
+}
+
+/*
+ * dclist_move_head
+ * Move 'node' from its current position in the list to the head position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_head(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ dlist_move_head(&head->dlist, node);
+}
+
+/*
+ * dclist_move_tail
+ * Move 'node' from its current position in the list to the tail position
+ * in 'head'.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline void
+dclist_move_tail(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ dlist_move_tail(&head->dlist, node);
+}
+
+/*
+ * dclist_has_next
+ * Check whether 'node' has a following node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_next(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ return dlist_has_next(&head->dlist, node);
+}
+
+/*
+ * dclist_has_prev
+ * Check whether 'node' has a preceding node.
+ *
+ * Caution: 'node' must be a member of 'head'.
+ */
+static inline bool
+dclist_has_prev(dclist_head *head, dlist_node *node)
+{
+ dlist_member_check(&head->dlist, node);
+ Assert(head->count > 0);
+
+ return dlist_has_prev(&head->dlist, node);
+}
+
+/*
+ * dclist_next_node
+ * Return the next node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_next_node(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+
+ return dlist_next_node(&head->dlist, node);
+}
+
+/*
+ * dclist_prev_node
+ * Return the prev node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_prev_node(dclist_head *head, dlist_node *node)
+{
+ Assert(head->count > 0);
+
+ return dlist_prev_node(&head->dlist, node);
+}
+
+/* internal support function to get address of head element's struct */
+static inline void *
+dclist_head_element_off(dclist_head *head, size_t off)
+{
+ Assert(!dclist_is_empty(head));
+
+ return (char *) head->dlist.head.next - off;
+}
+
+/*
+ * dclist_head_node
+ * Return the first node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_head_node(dclist_head *head)
+{
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_head_element_off(&head->dlist, 0);
+}
+
+/* internal support function to get address of tail element's struct */
+static inline void *
+dclist_tail_element_off(dclist_head *head, size_t off)
+{
+ Assert(!dclist_is_empty(head));
+
+ return (char *) head->dlist.head.prev - off;
+}
+
+/*
+ * Return the last node in the list (there must be one).
+ */
+static inline dlist_node *
+dclist_tail_node(dclist_head *head)
+{
+ Assert(head->count > 0);
+
+ return (dlist_node *) dlist_tail_element_off(&head->dlist, 0);
+}
+
+/*
+ * dclist_count
+ * Returns the stored number of entries in 'head'
+ */
+static inline uint32
+dclist_count(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+
+ return head->count;
+}
+
+/*
+ * Return the containing struct of 'type' where 'membername' is the dlist_node
+ * pointed at by 'ptr'.
+ *
+ * This is used to convert a dlist_node * back to its containing struct.
+ *
+ * Note: This is effectively just the same as dlist_container, so reuse that.
+ */
+#define dclist_container(type, membername, ptr) \
+ dlist_container(type, membername, ptr)
+
+ /*
+ * Return the address of the first element in the list.
+ *
+ * The list must not be empty.
+ */
+#define dclist_head_element(type, membername, lhead) \
+ (AssertVariableIsOfTypeMacro(((type *) NULL)->membername, dlist_node), \
+ (type *) dclist_head_element_off(lhead, offsetof(type, membername)))
+
+ /*
+ * Return the address of the last element in the list.
+ *
+ * The list must not be empty.
+ */
+#define dclist_tail_element(type, membername, lhead) \
+ (AssertVariableIsOfTypeMacro(((type *) NULL)->membername, dlist_node), \
+ ((type *) dclist_tail_element_off(lhead, offsetof(type, membername))))
+
+
+/* Iterators for dclists */
+#define dclist_foreach(iter, lhead) \
+ dlist_foreach(iter, &((lhead)->dlist))
+
+#define dclist_foreach_modify(iter, lhead) \
+ dlist_foreach_modify(iter, &((lhead)->dlist))
+
+#define dclist_reverse_foreach(iter, lhead) \
+ dlist_reverse_foreach(iter, &((lhead)->dlist))
/* singly linked list implementation */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 02b59a1931..b23d8cc4f9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -534,8 +534,7 @@ struct ReorderBuffer
/*
* Transactions and subtransactions that have modified system catalogs.
*/
- dlist_head catchange_txns;
- int catchange_ntxns;
+ dclist_head catchange_txns;
/*
* one-entry sized cache for by_txn. Very frequently the same txn gets
diff --git a/src/include/utils/pgstat_internal.h b/src/include/utils/pgstat_internal.h
index c869533b28..e2c7b59324 100644
--- a/src/include/utils/pgstat_internal.h
+++ b/src/include/utils/pgstat_internal.h
@@ -162,8 +162,7 @@ typedef struct PgStat_SubXactStatus
* if the transaction commits/aborts. To handle replicas and crashes,
* stats drops are included in commit / abort records.
*/
- dlist_head pending_drops;
- int pending_drops_count;
+ dclist_head pending_drops;
/*
* Tuple insertion/deletion counts for an open transaction can't be
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2f02cc8f42..c15f990cbb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3194,6 +3194,7 @@ datapagemap_t
dateKEY
datetkn
dce_uuid_t
+dclist_head
decimal
deparse_columns
deparse_context
On Mon, Oct 31, 2022 at 8:26 AM David Rowley <dgrowleyml@gmail.com> wrote:
I looked at dlist_check() and I didn't quite manage to figure out why
the cast is needed. As far as I can see, there are no calls where we
only pass dlist_head solely for the dlist_check(). For
dlist_member_check(), dlist_delete_from() does not use the 'head'
parameter for anything apart from dlist_member_check(), so I believe
the cast is required for 'head'. I think I'd rather only add the cast
for 'node' unless we really require it. Cargo-culting it in there just
because that's what the other macros do does not seem like a good idea
to me.
Hm, you're right, dlist_member_check() needs it. Also, slist_check()
needs it for slist_has_next(). dlist_check() doesn't need it, however,
keeping it intact doesn't harm, I guess.
My original thoughts were that it seems unlikely we'd ever give an
assert build a workload that would ever have 2^32 dlist_nodes in
dclist. Having said that, perhaps it would do no harm to add some
overflow checks to 'count'. I've gone and added some
Assert(head->count > 0) after we do count++.
So, when an overflow occurs, the head->count wraps around after
PG_UINT32_MAX, meaning, becomes 0 and we will catch it in an assert
build. This looks reasonable to me. However, the responsibility lies
with the developers to deal with such overflows.
8. Wondering if we need dlist_delete_from() at all. Can't we just add
dlist_member_check() dclist_delete_from() and call dlist_delete()
directly?Certainly, but I made it that way on purpose. I wanted dclist to have
a superset of the functions that dlist has. I just see no reason why
dlist shouldn't have dlist_delete_from() when dclist has it.
Okay.
I've attached the v3 version of the patch which includes some
additional polishing work.
Thanks. The v3 patch looks good to me.
BTW, do we need sclist_* as well? AFAICS, no such use-case exists
needing slist and element count, maybe we don't need.
I'm wondering if adding count to dlist_head and maintaining it as part
of the existing dlist_* data structure and functions is any better
than introducing dclist_*? In that case, we need only one function,
dlist_count, no? Or do we choose to go dclist_* because we want to
avoid the extra cost of maintaining count within dlist_*? If yes, is
maintaining count in dlist_* really costly?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Mon, 31 Oct 2022 at 19:05, Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
So, when an overflow occurs, the head->count wraps around after
PG_UINT32_MAX, meaning, becomes 0 and we will catch it in an assert
build. This looks reasonable to me. However, the responsibility lies
with the developers to deal with such overflows.
I'm not sure what the alternatives are here. One of the advantages of
dlist over List is that there are no memory allocations to add a node
which is already allocated onto a list. lappend() might need to
enlarge the list, which means you can't do that in a critical section.
It's currently OK to add an item to a dlist in a critical section,
however, if we add an elog(ERROR) then it won't be. The best I think
we can do is to just let the calling code ensure that it only uses
dlist when it's certain that there can't be more than 2^32 items to
store at once.
Additionally, everywhere I've replaced dlist with dclist in the patch
used either an int or uint32 for the counter. There was no code which
checked if the existing counter had wrapped.
Thanks. The v3 patch looks good to me.
Great. Thanks for having a look.
BTW, do we need sclist_* as well? AFAICS, no such use-case exists
needing slist and element count, maybe we don't need.
I don't see anywhere that requires it.
I'm wondering if adding count to dlist_head and maintaining it as part
of the existing dlist_* data structure and functions is any better
than introducing dclist_*? In that case, we need only one function,
dlist_count, no? Or do we choose to go dclist_* because we want to
avoid the extra cost of maintaining count within dlist_*? If yes, is
maintaining count in dlist_* really costly?
I have a few reasons for not wanting to do that:
1) I think dlist operations are very fast at the moment. The fact
that the functions are static inline tells me the function call
overhead matters. Therefore, it's likely maintaining a count also
matters.
2) Code bloat. The functions are static inline. That means all
compiled code that adds or removes an item from a dlist would end up
larger. That results in more instruction cache misses.
3) I've no reason to believe that all call sites that do
dlist_delete() have the ability to know which list the node is on.
Just look at ReorderBufferAssignChild(). I decided to not convert the
subtxns dlist into a dclist as the subtransaction sometimes seems to
go onto the top transaction list before it's moved to the sub-txn's
list.
4) There's very little or no scope for bugs in dclist relating to the
dlist implementation as all that stuff is done by just calling the
dlist_* functions. The only scope is really that it could call the
wrong dlist_* function. It does not seem terribly hard to ensure we
don't write any bugs like that.
David
On Mon, Oct 31, 2022 at 12:44 PM David Rowley <dgrowleyml@gmail.com> wrote:
On Mon, 31 Oct 2022 at 19:05, Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:So, when an overflow occurs, the head->count wraps around after
PG_UINT32_MAX, meaning, becomes 0 and we will catch it in an assert
build. This looks reasonable to me. However, the responsibility lies
with the developers to deal with such overflows.I'm not sure what the alternatives are here. One of the advantages of
dlist over List is that there are no memory allocations to add a node
which is already allocated onto a list. lappend() might need to
enlarge the list, which means you can't do that in a critical section.
Using uint64 is one option to allow many elements, however, I'm also
fine with removing the overflow assertions Assert(head->count > 0);
/* count overflow check */ altogether and let the callers take the
responsibility. dlist_* and dclist_* callers already have another
responsibility - Caution: 'foo' must be a member of 'head'.
It's currently OK to add an item to a dlist in a critical section,
however, if we add an elog(ERROR) then it won't be.
dlist_check() and dlist_member_check() have an elog(ERROR) and the
above statement isn't true in case of ILIST_DEBUG-defined builds.
The best I think
we can do is to just let the calling code ensure that it only uses
dlist when it's certain that there can't be more than 2^32 items to
store at once.
Right.
+ * able to store a maximum of PG_UINT32_MAX elements. It is up to the caller
+ * to ensure no more than this many items are added to a dclist.
The above comment seems fine to me, if we really want to enforce any
overflow checks on non-debug, non-assert builds, it might add some
costs to dclist_* functions.
I'm wondering if adding count to dlist_head and maintaining it as part
of the existing dlist_* data structure and functions is any better
than introducing dclist_*? In that case, we need only one function,
dlist_count, no? Or do we choose to go dclist_* because we want to
avoid the extra cost of maintaining count within dlist_*? If yes, is
maintaining count in dlist_* really costly?I have a few reasons for not wanting to do that:
1) I think dlist operations are very fast at the moment. The fact
that the functions are static inline tells me the function call
overhead matters. Therefore, it's likely maintaining a count also
matters.
2) Code bloat. The functions are static inline. That means all
compiled code that adds or removes an item from a dlist would end up
larger. That results in more instruction cache misses.
This seems a fair point to me.
3) I've no reason to believe that all call sites that do
dlist_delete() have the ability to know which list the node is on.
Just look at ReorderBufferAssignChild(). I decided to not convert the
subtxns dlist into a dclist as the subtransaction sometimes seems to
go onto the top transaction list before it's moved to the sub-txn's
list.
4) There's very little or no scope for bugs in dclist relating to the
dlist implementation as all that stuff is done by just calling the
dlist_* functions. The only scope is really that it could call the
wrong dlist_* function. It does not seem terribly hard to ensure we
don't write any bugs like that.
Right.
Thanks. The v3 patch looks good to me.
Great. Thanks for having a look.
I will take another look at v3 tomorrow and probably mark it RfC.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi hackers,
I will take another look at v3 tomorrow and probably mark it RfC.
I very much like the patch. While on it:
```
+static inline bool
+dclist_is_empty(dclist_head *head)
+{
+ Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
+ return (head->count == 0);
+}
```
Should we consider const'ifying the arguments of the dlist_*/dclist_*
functions that don't change the arguments?
Additionally it doesn't seem that we have any unit tests for dlist /
dclist. Should we consider adding unit tests for them to
src/test/regress?
To clarify, IMO both questions are out of scope of this specific patch
and should be submitted separately.
--
Best regards,
Aleksander Alekseev
On Mon, Oct 31, 2022 at 6:58 PM Aleksander Alekseev
<aleksander@timescale.com> wrote:
Hi hackers,
I will take another look at v3 tomorrow and probably mark it RfC.
I very much like the patch. While on it:
``` +static inline bool +dclist_is_empty(dclist_head *head) +{ + Assert(dlist_is_empty(&head->dlist) == (head->count == 0)); + return (head->count == 0); +} ```Should we consider const'ifying the arguments of the dlist_*/dclist_*
functions that don't change the arguments?
+1, but as a separate discussion/thread/patch IMO.
Additionally it doesn't seem that we have any unit tests for dlist /
dclist. Should we consider adding unit tests for them to
src/test/regress?
Most of the dlist_* functions are being covered I guess. AFAICS,
dclist_* functions that aren't covered are dclist_insert_after(),
dclist_insert_before(), dclist_pop_head_node(), dclist_move_head(),
dclist_move_tail(), dclist_has_next(), dclist_has_prev(),
dclist_next_node(), dclist_prev_node(), dclist_head_element_off(),
dclist_head_node(), dclist_tail_element_off(), dclist_head_element().
IMO, adding an extension under src/test/modules to cover missing or
all dlist_* and dclist_* functions makes sense. It improves the code
coverage. FWIW, test_lfind is one such recent test extension.
To clarify, IMO both questions are out of scope of this specific patch
and should be submitted separately.
You're right, both of them must be discussed separately.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Mon, Oct 31, 2022 at 8:28 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
On Mon, Oct 31, 2022 at 6:58 PM Aleksander Alekseev
<aleksander@timescale.com> wrote:Hi hackers,
I will take another look at v3 tomorrow and probably mark it RfC.
I very much like the patch. While on it:
I took another look at v3 patch today and it looked good to me, hence
marked it RfC - https://commitfest.postgresql.org/40/3967/
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Tue, 1 Nov 2022 at 20:55, Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
I took another look at v3 patch today and it looked good to me, hence
marked it RfC - https://commitfest.postgresql.org/40/3967/
Many thanks for reviewing this.
If nobody has any objections, I plan to push this tomorrow morning New
Zealand time (around 10 hours from now).
David
On Tue, 1 Nov 2022 at 23:19, David Rowley <dgrowleyml@gmail.com> wrote:
If nobody has any objections, I plan to push this tomorrow morning New
Zealand time (around 10 hours from now).
Pushed. Thank you both for reviewing this.
David
Hi David,
Pushed. Thank you both for reviewing this.
Thanks for applying the patch.
Should we consider const'ifying the arguments of the dlist_*/dclist_*
functions that don't change the arguments?
[...]You're right, both of them must be discussed separately.
I would like to piggyback on this thread to propose the const'ifying
patch, if that's OK. Here it is.
--
Best regards,
Aleksander Alekseev
Attachments:
v1-0001-Constify-the-arguments-of-ilist.c-h-functions.patchapplication/octet-stream; name=v1-0001-Constify-the-arguments-of-ilist.c-h-functions.patchDownload
From 2aab14a052de2d09ee7171cfffe94aa88175c72d Mon Sep 17 00:00:00 2001
From: Aleksander Alekseev <aleksander@timescale.com>
Date: Wed, 2 Nov 2022 11:26:23 +0300
Subject: [PATCH v1] Constify the arguments of ilist.c/h functions
Const qualifiers ensure that we don't do something stupid in the function
implementation. Additionally they clarify the interface. As an example:
void
slist_delete(slist_head *head, const slist_node *node)
Here one can instantly tell that node->next is not going to be set to NULL.
Finally, const qualifiers potentially allow the compiler to do more
optimizations. This being said no benchmarking was done for this patch.
Author: Aleksander Alekseev
Reviewed-by: Bharath Rupireddy, David Rowley
Discussion: https://postgr.es/m/CAApHDvrtVxr+FXEX0VbViCFKDGxA3tWDgw9oFewNXCJMmwLjLg@mail.gmail.com
---
src/backend/lib/ilist.c | 8 +++---
src/include/lib/ilist.h | 56 ++++++++++++++++++++---------------------
2 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/src/backend/lib/ilist.c b/src/backend/lib/ilist.c
index e8ea981176..a1f236a9e1 100644
--- a/src/backend/lib/ilist.c
+++ b/src/backend/lib/ilist.c
@@ -28,7 +28,7 @@
* Caution: this is O(n); consider using slist_delete_current() instead.
*/
void
-slist_delete(slist_head *head, slist_node *node)
+slist_delete(slist_head *head, const slist_node *node)
{
slist_node *last = &head->head;
slist_node *cur;
@@ -57,7 +57,7 @@ slist_delete(slist_head *head, slist_node *node)
* Validate that 'node' is a member of 'head'
*/
void
-dlist_member_check(dlist_head *head, dlist_node *node)
+dlist_member_check(const dlist_head *head, const dlist_node *node)
{
dlist_iter iter;
@@ -73,7 +73,7 @@ dlist_member_check(dlist_head *head, dlist_node *node)
* Verify integrity of a doubly linked list
*/
void
-dlist_check(dlist_head *head)
+dlist_check(const dlist_head *head)
{
dlist_node *cur;
@@ -110,7 +110,7 @@ dlist_check(dlist_head *head)
* Verify integrity of a singly linked list
*/
void
-slist_check(slist_head *head)
+slist_check(const slist_head *head)
{
slist_node *cur;
diff --git a/src/include/lib/ilist.h b/src/include/lib/ilist.h
index 3c543e7c36..6c2b05721b 100644
--- a/src/include/lib/ilist.h
+++ b/src/include/lib/ilist.h
@@ -286,12 +286,12 @@ typedef struct slist_mutable_iter
/* Prototypes for functions too big to be inline */
/* Caution: this is O(n); consider using slist_delete_current() instead */
-extern void slist_delete(slist_head *head, slist_node *node);
+extern void slist_delete(slist_head *head, const slist_node *node);
#ifdef ILIST_DEBUG
-extern void dlist_member_check(dlist_head *head, dlist_node *node);
-extern void dlist_check(dlist_head *head);
-extern void slist_check(slist_head *head);
+extern void dlist_member_check(const dlist_head *head, const dlist_node *node);
+extern void dlist_check(const dlist_head *head);
+extern void slist_check(const slist_head *head);
#else
/*
* These seemingly useless casts to void are here to keep the compiler quiet
@@ -322,7 +322,7 @@ dlist_init(dlist_head *head)
* An empty list has either its first 'next' pointer set to NULL, or to itself.
*/
static inline bool
-dlist_is_empty(dlist_head *head)
+dlist_is_empty(const dlist_head *head)
{
dlist_check(head);
@@ -465,7 +465,7 @@ dlist_move_tail(dlist_head *head, dlist_node *node)
* Caution: unreliable if 'node' is not in the list.
*/
static inline bool
-dlist_has_next(dlist_head *head, dlist_node *node)
+dlist_has_next(const dlist_head *head, const dlist_node *node)
{
return node->next != &head->head;
}
@@ -475,7 +475,7 @@ dlist_has_next(dlist_head *head, dlist_node *node)
* Caution: unreliable if 'node' is not in the list.
*/
static inline bool
-dlist_has_prev(dlist_head *head, dlist_node *node)
+dlist_has_prev(const dlist_head *head, const dlist_node *node)
{
return node->prev != &head->head;
}
@@ -484,7 +484,7 @@ dlist_has_prev(dlist_head *head, dlist_node *node)
* Return the next node in the list (there must be one).
*/
static inline dlist_node *
-dlist_next_node(dlist_head *head, dlist_node *node)
+dlist_next_node(const dlist_head *head, const dlist_node *node)
{
Assert(dlist_has_next(head, node));
return node->next;
@@ -494,7 +494,7 @@ dlist_next_node(dlist_head *head, dlist_node *node)
* Return previous node in the list (there must be one).
*/
static inline dlist_node *
-dlist_prev_node(dlist_head *head, dlist_node *node)
+dlist_prev_node(const dlist_head *head, const dlist_node *node)
{
Assert(dlist_has_prev(head, node));
return node->prev;
@@ -502,7 +502,7 @@ dlist_prev_node(dlist_head *head, dlist_node *node)
/* internal support function to get address of head element's struct */
static inline void *
-dlist_head_element_off(dlist_head *head, size_t off)
+dlist_head_element_off(const dlist_head *head, size_t off)
{
Assert(!dlist_is_empty(head));
return (char *) head->head.next - off;
@@ -512,14 +512,14 @@ dlist_head_element_off(dlist_head *head, size_t off)
* Return the first node in the list (there must be one).
*/
static inline dlist_node *
-dlist_head_node(dlist_head *head)
+dlist_head_node(const dlist_head *head)
{
return (dlist_node *) dlist_head_element_off(head, 0);
}
/* internal support function to get address of tail element's struct */
static inline void *
-dlist_tail_element_off(dlist_head *head, size_t off)
+dlist_tail_element_off(const dlist_head *head, size_t off)
{
Assert(!dlist_is_empty(head));
return (char *) head->head.prev - off;
@@ -529,7 +529,7 @@ dlist_tail_element_off(dlist_head *head, size_t off)
* Return the last node in the list (there must be one).
*/
static inline dlist_node *
-dlist_tail_node(dlist_head *head)
+dlist_tail_node(const dlist_head *head)
{
return (dlist_node *) dlist_tail_element_off(head, 0);
}
@@ -629,7 +629,7 @@ dclist_init(dclist_head *head)
* Returns true if the list is empty, otherwise false.
*/
static inline bool
-dclist_is_empty(dclist_head *head)
+dclist_is_empty(const dclist_head *head)
{
Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
return (head->count == 0);
@@ -773,7 +773,7 @@ dclist_move_tail(dclist_head *head, dlist_node *node)
* Caution: 'node' must be a member of 'head'.
*/
static inline bool
-dclist_has_next(dclist_head *head, dlist_node *node)
+dclist_has_next(const dclist_head *head, const dlist_node *node)
{
dlist_member_check(&head->dlist, node);
Assert(head->count > 0);
@@ -788,7 +788,7 @@ dclist_has_next(dclist_head *head, dlist_node *node)
* Caution: 'node' must be a member of 'head'.
*/
static inline bool
-dclist_has_prev(dclist_head *head, dlist_node *node)
+dclist_has_prev(const dclist_head *head, const dlist_node *node)
{
dlist_member_check(&head->dlist, node);
Assert(head->count > 0);
@@ -801,7 +801,7 @@ dclist_has_prev(dclist_head *head, dlist_node *node)
* Return the next node in the list (there must be one).
*/
static inline dlist_node *
-dclist_next_node(dclist_head *head, dlist_node *node)
+dclist_next_node(const dclist_head *head, const dlist_node *node)
{
Assert(head->count > 0);
@@ -813,7 +813,7 @@ dclist_next_node(dclist_head *head, dlist_node *node)
* Return the prev node in the list (there must be one).
*/
static inline dlist_node *
-dclist_prev_node(dclist_head *head, dlist_node *node)
+dclist_prev_node(const dclist_head *head, const dlist_node *node)
{
Assert(head->count > 0);
@@ -822,7 +822,7 @@ dclist_prev_node(dclist_head *head, dlist_node *node)
/* internal support function to get address of head element's struct */
static inline void *
-dclist_head_element_off(dclist_head *head, size_t off)
+dclist_head_element_off(const dclist_head *head, size_t off)
{
Assert(!dclist_is_empty(head));
@@ -834,7 +834,7 @@ dclist_head_element_off(dclist_head *head, size_t off)
* Return the first node in the list (there must be one).
*/
static inline dlist_node *
-dclist_head_node(dclist_head *head)
+dclist_head_node(const dclist_head *head)
{
Assert(head->count > 0);
@@ -843,7 +843,7 @@ dclist_head_node(dclist_head *head)
/* internal support function to get address of tail element's struct */
static inline void *
-dclist_tail_element_off(dclist_head *head, size_t off)
+dclist_tail_element_off(const dclist_head *head, size_t off)
{
Assert(!dclist_is_empty(head));
@@ -854,7 +854,7 @@ dclist_tail_element_off(dclist_head *head, size_t off)
* Return the last node in the list (there must be one).
*/
static inline dlist_node *
-dclist_tail_node(dclist_head *head)
+dclist_tail_node(const dclist_head *head)
{
Assert(head->count > 0);
@@ -866,7 +866,7 @@ dclist_tail_node(dclist_head *head)
* Returns the stored number of entries in 'head'
*/
static inline uint32
-dclist_count(dclist_head *head)
+dclist_count(const dclist_head *head)
{
Assert(dlist_is_empty(&head->dlist) == (head->count == 0));
@@ -929,7 +929,7 @@ slist_init(slist_head *head)
* Is the list empty?
*/
static inline bool
-slist_is_empty(slist_head *head)
+slist_is_empty(const slist_head *head)
{
slist_check(head);
@@ -977,7 +977,7 @@ slist_pop_head_node(slist_head *head)
* Check whether 'node' has a following node.
*/
static inline bool
-slist_has_next(slist_head *head, slist_node *node)
+slist_has_next(const slist_head *head, const slist_node *node)
{
slist_check(head);
@@ -988,7 +988,7 @@ slist_has_next(slist_head *head, slist_node *node)
* Return the next node in the list (there must be one).
*/
static inline slist_node *
-slist_next_node(slist_head *head, slist_node *node)
+slist_next_node(const slist_head *head, const slist_node *node)
{
Assert(slist_has_next(head, node));
return node->next;
@@ -996,7 +996,7 @@ slist_next_node(slist_head *head, slist_node *node)
/* internal support function to get address of head element's struct */
static inline void *
-slist_head_element_off(slist_head *head, size_t off)
+slist_head_element_off(const slist_head *head, size_t off)
{
Assert(!slist_is_empty(head));
return (char *) head->head.next - off;
@@ -1006,7 +1006,7 @@ slist_head_element_off(slist_head *head, size_t off)
* Return the first node in the list (there must be one).
*/
static inline slist_node *
-slist_head_node(slist_head *head)
+slist_head_node(const slist_head *head)
{
return (slist_node *) slist_head_element_off(head, 0);
}
--
2.38.0
On Wed, Nov 2, 2022 at 2:23 PM Aleksander Alekseev
<aleksander@timescale.com> wrote:
Hi David,
Pushed. Thank you both for reviewing this.
Thanks for applying the patch.
Should we consider const'ifying the arguments of the dlist_*/dclist_*
functions that don't change the arguments?
[...]You're right, both of them must be discussed separately.
I would like to piggyback on this thread to propose the const'ifying
patch, if that's OK. Here it is.
Thanks for the patch. IMO, this can be discussed in a separate thread
to get more thoughts from the hackers.
BTW, there's proclist_* data structure which might need the similar
const'ifying.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi Bharath,
Thanks for the patch. IMO, this can be discussed in a separate thread
to get more thoughts from the hackers.
OK, I started a new thread [1]/messages/by-id/CAJ7c6TM2=08mNKD9aJg8vEY9hd+G4L7+Nvh30UiNT3kShgRgNg@mail.gmail.com, thanks.
Regarding the improvement of the code coverage I realized that this
may be a good patch for a newcomer. I know several people who may be
interested in starting to contribute to PostgreSQL. Maybe I'll be able
to find a volunteer.
[1]: /messages/by-id/CAJ7c6TM2=08mNKD9aJg8vEY9hd+G4L7+Nvh30UiNT3kShgRgNg@mail.gmail.com
--
Best regards,
Aleksander Alekseev
On Mon, Nov 7, 2022 at 2:37 PM Aleksander Alekseev
<aleksander@timescale.com> wrote:
Regarding the improvement of the code coverage I realized that this
may be a good patch for a newcomer. I know several people who may be
interested in starting to contribute to PostgreSQL. Maybe I'll be able
to find a volunteer.
Hm. Or adding a ToDo item here https://wiki.postgresql.org/wiki/Todo
might also help?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi Bharath,
Regarding the improvement of the code coverage I realized that this
may be a good patch for a newcomer. I know several people who may be
interested in starting to contribute to PostgreSQL. Maybe I'll be able
to find a volunteer.Hm. Or adding a ToDo item here https://wiki.postgresql.org/wiki/Todo
might also help?
Good point. Will it be better to use the "Miscellaneous Other" section
for this or create a new "Code coverage" section?
--
Best regards,
Aleksander Alekseev