change on_exit_nicely_list array to the dynamic array to increase slots at run time for pg_restore
Hi,
(refer file src/bin/pg_dump/pg_backup_utils.c)
While doing some code changes with pg_dumpall and pg_rsetore[1]pg_dumpall with non-text mode </messages/by-id/cb103623-8ee6-4ba5-a2c9-f32e3a4933fa@dunslane.net>, we noticed
that on_exit_nicely_list array has only fixed slots (MAX_ON_EXIT_NICELY=20)
but in some cases, we need more slots for pg_restore.
*Ex: restore more than 20 databases by single pg_restore command.*
We are working on a patch[1]pg_dumpall with non-text mode </messages/by-id/cb103623-8ee6-4ba5-a2c9-f32e3a4933fa@dunslane.net> which dumps all the databases in non-text mode
by pg_dumpall and then we are restoring those dumps by pg_restore. So we
need more slots due to multiple databases.
Apart from the attached patch solution, we thought of some more solutions.
*Solution 1*: reset array index with each database restore, but this might
break some other things.
*Solution 2:* for each database, we can keep the index of the
on_exit_nicely_list array and after restoring a particular database we can
reset the index to old value but this looks like a hack.
Here, I am proposing a patch which will dynamically enlarge the
on_exit_nicely_list array by doubling the previous size.
Thoughts?
[1]: pg_dumpall with non-text mode </messages/by-id/cb103623-8ee6-4ba5-a2c9-f32e3a4933fa@dunslane.net>
</messages/by-id/cb103623-8ee6-4ba5-a2c9-f32e3a4933fa@dunslane.net>
[2]: problem of on_exit_nicely_list array <202503051512.tr7vcirldfns@alvherre.pgsql> -- Thanks and Regards Mahendra Singh Thalor EnterpriseDB: http://www.enterprisedb.com
<202503051512.tr7vcirldfns@alvherre.pgsql>
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v01-enlarge-on_exit_nicely_list-array-whenever-no-slot.patchapplication/octet-stream; name=v01-enlarge-on_exit_nicely_list-array-whenever-no-slot.patchDownload
From d70aa4c127fbcecb11e61393c17735a1e3473f86 Mon Sep 17 00:00:00 2001
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Date: Mon, 10 Mar 2025 13:55:15 +0530
Subject: [PATCH] enlarge on_exit_nicely_list array whenever no slot left
Earlier we were using fixed size array with the size of MAX_ON_EXIT_NICELY=20
but for pg_restore, we need more slots as we will restore multiple database
by single dump file of pg_dumpall.
Instead of fixed size array, now we will allocate memory dynamically and
if there is no slot, then we will double the previous size.
---
src/bin/pg_dump/pg_backup_utils.c | 42 +++++++++++++++++++++++++++----
src/tools/pgindent/typedefs.list | 1 +
2 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/src/bin/pg_dump/pg_backup_utils.c b/src/bin/pg_dump/pg_backup_utils.c
index 79aec5f5158..cecec14982d 100644
--- a/src/bin/pg_dump/pg_backup_utils.c
+++ b/src/bin/pg_dump/pg_backup_utils.c
@@ -23,13 +23,16 @@ const char *progname = NULL;
#define MAX_ON_EXIT_NICELY 20
-static struct
+typedef struct on_exit_nicely_item
{
on_exit_nicely_callback function;
void *arg;
-} on_exit_nicely_list[MAX_ON_EXIT_NICELY];
+} on_exit_nicely_item;
-static int on_exit_nicely_index;
+static on_exit_nicely_item *on_exit_nicely_list = NULL;
+static int on_exit_nicely_index;
+static int max_on_exit_nicely_list;
+static void EnlargeExitNicelyArray(void);
/*
* Parse a --section=foo command line argument.
@@ -59,13 +62,42 @@ set_dump_section(const char *arg, int *dumpSections)
}
}
+/*
+ * EnlargeExitNicelyArray
+ *
+ * Enlarges the array size, if required. If first time, we allocate the memory
+ * using malloc.
+ */
+static void
+EnlargeExitNicelyArray(void)
+{
+ /* If array has space for 1 entry, then return from here. */
+ if (on_exit_nicely_index < max_on_exit_nicely_list)
+ return;
+
+ /* Double the capacity of the array. */
+ max_on_exit_nicely_list = (max_on_exit_nicely_list > 0) ? max_on_exit_nicely_list * 2 : MAX_ON_EXIT_NICELY;
+
+ /* If array is pointing to NULL, then allocate memory for fixed size. */
+ if (!on_exit_nicely_list)
+ on_exit_nicely_list = (on_exit_nicely_item *) pg_malloc0(max_on_exit_nicely_list * sizeof(on_exit_nicely_item));
+ else
+ {
+ /*
+ * As array already have some memory so instead of alloc, using
+ * repalloc so that all old data of array will be copied into new
+ * allocated memory.
+ */
+ on_exit_nicely_list = (on_exit_nicely_item *) pg_realloc(on_exit_nicely_list,
+ max_on_exit_nicely_list * sizeof(on_exit_nicely_item));
+ }
+}
/* Register a callback to be run when exit_nicely is invoked. */
void
on_exit_nicely(on_exit_nicely_callback function, void *arg)
{
- if (on_exit_nicely_index >= MAX_ON_EXIT_NICELY)
- pg_fatal("out of on_exit_nicely slots");
+ EnlargeExitNicelyArray();
on_exit_nicely_list[on_exit_nicely_index].function = function;
on_exit_nicely_list[on_exit_nicely_index].arg = arg;
on_exit_nicely_index++;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..da458d1cb6a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3724,6 +3724,7 @@ oidKEY
oidvector
on_dsm_detach_callback
on_exit_nicely_callback
+on_exit_nicely_item
openssl_tls_init_hook_typ
ossl_EVP_cipher_func
other
--
2.39.3
Mahendra Singh Thalor <mahi6run@gmail.com> writes:
While doing some code changes with pg_dumpall and pg_rsetore[1], we noticed
that on_exit_nicely_list array has only fixed slots (MAX_ON_EXIT_NICELY=20)
but in some cases, we need more slots for pg_restore.
*Ex: restore more than 20 databases by single pg_restore command.*
We are working on a patch[1] which dumps all the databases in non-text mode
by pg_dumpall and then we are restoring those dumps by pg_restore. So we
need more slots due to multiple databases.
I can't help thinking that this means you're doing it wrong.
That on_exit infrastructure is quite rickety --- see the comments for
exit_nicely() itself --- so I don't think I trust the idea of
dynamically adding callbacks. In the present usage there is actually
only *one* callback and it's registered very near the start of either
pg_dump or pg_restore. That avoids the platform-dependent hazards
mentioned in the comment, but adding more callbacks later might not.
I think you'd be better off tracking the cleanup work you have to do
in some other place.
regards, tom lane
On Mon, Mar 10, 2025 at 2:24 PM Mahendra Singh Thalor
<mahi6run@gmail.com> wrote:
Hi,
(refer file src/bin/pg_dump/pg_backup_utils.c)While doing some code changes with pg_dumpall and pg_rsetore[1], we noticed that on_exit_nicely_list array has only fixed slots (MAX_ON_EXIT_NICELY=20) but in some cases, we need more slots for pg_restore.
Ex: restore more than 20 databases by single pg_restore command.We are working on a patch[1] which dumps all the databases in non-text mode by pg_dumpall and then we are restoring those dumps by pg_restore. So we need more slots due to multiple databases.
Apart from the attached patch solution, we thought of some more solutions.
Solution 1: reset array index with each database restore, but this might break some other things.
Solution 2: for each database, we can keep the index of the on_exit_nicely_list array and after restoring a particular database we can reset the index to old value but this looks like a hack.Here, I am proposing a patch which will dynamically enlarge the on_exit_nicely_list array by doubling the previous size.
Thoughts?
This reply might belong in another thread, but since you raised the
issue here, instead of registering multiple functions with
exit_nicely, why not modify the argument of the exit function
(archive_close_connection()) to handle multiple arguments? In short,
you could change ShutdownInformation[1]typedef struct ShutdownInformation { ParallelState *pstate; Archive *AHX; -> change this to List of Archive* } ShutdownInformation; so that, instead of holding a
single "Archive," it holds a list, allowing you to append different
Archive handles to the list.
[1]: typedef struct ShutdownInformation { ParallelState *pstate; Archive *AHX; -> change this to List of Archive* } ShutdownInformation;
typedef struct ShutdownInformation
{
ParallelState *pstate;
Archive *AHX; -> change this to List of Archive*
} ShutdownInformation;
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, 11 Mar 2025 at 11:58, Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Mar 10, 2025 at 2:24 PM Mahendra Singh Thalor
<mahi6run@gmail.com> wrote:Hi,
(refer file src/bin/pg_dump/pg_backup_utils.c)While doing some code changes with pg_dumpall and pg_rsetore[1], we noticed that on_exit_nicely_list array has only fixed slots (MAX_ON_EXIT_NICELY=20) but in some cases, we need more slots for pg_restore.
Ex: restore more than 20 databases by single pg_restore command.We are working on a patch[1] which dumps all the databases in non-text mode by pg_dumpall and then we are restoring those dumps by pg_restore. So we need more slots due to multiple databases.
Apart from the attached patch solution, we thought of some more solutions.
Solution 1: reset array index with each database restore, but this might break some other things.
Solution 2: for each database, we can keep the index of the on_exit_nicely_list array and after restoring a particular database we can reset the index to old value but this looks like a hack.Here, I am proposing a patch which will dynamically enlarge the on_exit_nicely_list array by doubling the previous size.
Thoughts?
This reply might belong in another thread, but since you raised the
issue here, instead of registering multiple functions with
exit_nicely, why not modify the argument of the exit function
(archive_close_connection()) to handle multiple arguments? In short,
you could change ShutdownInformation[1] so that, instead of holding a
single "Archive," it holds a list, allowing you to append different
Archive handles to the list.[1]
typedef struct ShutdownInformation
{
ParallelState *pstate;
Archive *AHX; -> change this to List of Archive*
} ShutdownInformation;
Thanks Tom and Dilip for feedback.
As per above discussion, I did cleanup in different ways to fix the
problem in "non-text mde for pg_dumpall" patch.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com