Bump soft open file limit (RLIMIT_NOFILE) to hard limit on startup

From 8e964db585989734a5f6c1449ffb4c62e1190a6a Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 22:04:44 +0100
Subject: [PATCH v2 1/3] Bump pgbench soft open file limit (RLIMIT_NOFILE) to
 hard limit

The default open file limit of 1024 is extremely low, given modern
resources and kernel architectures. The reason that this hasn't changed
is because doing so would break legacy programs that use the select(2)
system call in hard to debug ways. So instead programs that want to
opt-in to a higher open file limit are expected to bump their soft limit
to their hard limit on startup. Details on this are very well explained
in a blogpost by the systemd author[1].

This starts bumping pgbench its soft open file limit to the hard open
file limit. This makes sure users are not told to change their ulimit,
and then retry. Instead we now do that for them automatically.

[1]: https://0pointer.net/blog/file-descriptor-limits.html
---
 src/bin/pgbench/pgbench.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 5e1fcf59c61..ea7bf980bea 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6815,6 +6815,15 @@ main(int argc, char **argv)
 #ifdef HAVE_GETRLIMIT
 				if (getrlimit(RLIMIT_NOFILE, &rlim) == -1)
 					pg_fatal("getrlimit failed: %m");
+
+				/*
+				 * Bump the soft limit to the hard limit to not run into low
+				 * file limits.
+				 */
+				rlim.rlim_cur = rlim.rlim_max;
+				if (setrlimit(RLIMIT_NOFILE, &rlim) == -1)
+					pg_fatal("setrlimit failed: %m");
+
 				if (rlim.rlim_cur < nclients + 3)
 				{
 					pg_log_error("need at least %d open files, but system limit is %ld",

base-commit: fd602f29c19d4f483f54d93abe240c12219d9f51
-- 
2.43.0

From e84092e3db7712db08dfc5f8824f692a53a1aa9e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v2 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml                 |  17 ++--
 src/backend/access/transam/xlogarchive.c |   4 +
 src/backend/archive/shell_archive.c      |   2 +
 src/backend/storage/file/fd.c            | 111 ++++++++++++++++++++---
 src/include/storage/fd.h                 |   2 +
 5 files changed, 116 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5e4f201e099..4886a02d47e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2352,15 +2352,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..bf17426039e 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,6 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
 
@@ -180,6 +181,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 
 	pgstat_report_wait_end();
 	pfree(xlogRestoreCmd);
+	RestoreCustomOpenFileLimit();
 
 	if (rc == 0)
 	{
@@ -325,10 +327,12 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 	rc = system(xlogRecoveryCmd);
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 
 	pfree(xlogRecoveryCmd);
 
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..c16f354a118 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -75,10 +75,12 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
 	rc = system(xlogarchcmd);
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 
 	if (rc != 0)
 	{
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..04fb93be56d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,12 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +951,82 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+#ifdef HAVE_GETRLIMIT
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+	struct rlimit rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+}
+#endif
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -969,7 +1051,6 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			j;
 
 #ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
 	int			getrlimit_status;
 #endif
 
@@ -977,9 +1058,11 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	fd = (int *) palloc(size * sizeof(int));
 
 #ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
+	getrlimit_status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
 	if (getrlimit_status != 0)
 		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+	else
+		custom_max_open_files = original_max_open_files;
 #endif							/* HAVE_GETRLIMIT */
 
 	/* dup until failure or probe limit reached */
@@ -993,13 +1076,21 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
+		if (getrlimit_status == 0 && highestfd >= custom_max_open_files.rlim_cur - 1)
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 #endif
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+#ifdef HAVE_GETRLIMIT
+			if (errno == ENFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+#endif
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1144,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -1078,6 +1164,7 @@ set_max_safe_fds(void)
 		 max_safe_fds, usable_fds, already_open);
 }
 
+
 /*
  * Open a file with BasicOpenFilePerm() and pass default file mode for the
  * fileMode parameter.
@@ -2724,6 +2811,7 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
@@ -2731,6 +2819,7 @@ TryAgain:
 	save_errno = errno;
 	pqsignal(SIGPIPE, SIG_IGN);
 	errno = save_errno;
+	RestoreCustomOpenFileLimit();
 	if (file != NULL)
 	{
 		AllocateDesc *desc = &allocatedDescs[numAllocatedDescs];
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From 44c196025cbf57a09f6aa1bc70646ed2537d9654 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v2 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 04fb93be56d..05c0792e4e1 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1149,6 +1149,9 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_files_per_process = max_safe_fds;
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From 761c3b0a161eb9d680354c9c20a872091ce224be Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v3 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 04fb93be56d..05c0792e4e1 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1149,6 +1149,9 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_files_per_process = max_safe_fds;
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From f3ed3a3484e1d3dcfea3d7f94cd548533a84995d Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v3 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml                 |  17 ++--
 src/backend/access/transam/xlogarchive.c |   4 +
 src/backend/archive/shell_archive.c      |   2 +
 src/backend/storage/file/fd.c            | 111 ++++++++++++++++++++---
 src/include/storage/fd.h                 |   2 +
 5 files changed, 116 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 336630ce417..6b4bf78a9fc 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2352,15 +2352,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..bf17426039e 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,6 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
 
@@ -180,6 +181,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 
 	pgstat_report_wait_end();
 	pfree(xlogRestoreCmd);
+	RestoreCustomOpenFileLimit();
 
 	if (rc == 0)
 	{
@@ -325,10 +327,12 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 	rc = system(xlogRecoveryCmd);
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 
 	pfree(xlogRecoveryCmd);
 
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..c16f354a118 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -75,10 +75,12 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
 	rc = system(xlogarchcmd);
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 
 	if (rc != 0)
 	{
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..04fb93be56d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,12 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +951,82 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+#ifdef HAVE_GETRLIMIT
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+	struct rlimit rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+}
+#endif
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -969,7 +1051,6 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			j;
 
 #ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
 	int			getrlimit_status;
 #endif
 
@@ -977,9 +1058,11 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	fd = (int *) palloc(size * sizeof(int));
 
 #ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
+	getrlimit_status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
 	if (getrlimit_status != 0)
 		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+	else
+		custom_max_open_files = original_max_open_files;
 #endif							/* HAVE_GETRLIMIT */
 
 	/* dup until failure or probe limit reached */
@@ -993,13 +1076,21 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
+		if (getrlimit_status == 0 && highestfd >= custom_max_open_files.rlim_cur - 1)
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 #endif
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+#ifdef HAVE_GETRLIMIT
+			if (errno == ENFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+#endif
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1144,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -1078,6 +1164,7 @@ set_max_safe_fds(void)
 		 max_safe_fds, usable_fds, already_open);
 }
 
+
 /*
  * Open a file with BasicOpenFilePerm() and pass default file mode for the
  * fileMode parameter.
@@ -2724,6 +2811,7 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
@@ -2731,6 +2819,7 @@ TryAgain:
 	save_errno = errno;
 	pqsignal(SIGPIPE, SIG_IGN);
 	errno = save_errno;
+	RestoreCustomOpenFileLimit();
 	if (file != NULL)
 	{
 		AllocateDesc *desc = &allocatedDescs[numAllocatedDescs];
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From fe11e75d913ac320f11ac2250c94fa58b118b2e1 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 22:04:44 +0100
Subject: [PATCH v3 1/3] Bump pgbench soft open file limit (RLIMIT_NOFILE) to
 hard limit

The default open file limit of 1024 is extremely low, given modern
resources and kernel architectures. The reason that this hasn't changed
is because doing so would break legacy programs that use the select(2)
system call in hard to debug ways. So instead programs that want to
opt-in to a higher open file limit are expected to bump their soft limit
to their hard limit on startup. Details on this are very well explained
in a blogpost by the systemd author[1].

This starts bumping pgbench its soft open file limit to the hard open
file limit. This makes sure users are not told to change their ulimit,
and then retry. Instead we now do that for them automatically.

[1]: https://0pointer.net/blog/file-descriptor-limits.html
---
 src/bin/pgbench/pgbench.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 5e1fcf59c61..8f53bf50ab1 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -6815,13 +6815,26 @@ main(int argc, char **argv)
 #ifdef HAVE_GETRLIMIT
 				if (getrlimit(RLIMIT_NOFILE, &rlim) == -1)
 					pg_fatal("getrlimit failed: %m");
-				if (rlim.rlim_cur < nclients + 3)
+
+				if (rlim.rlim_max < nclients + 3)
 				{
 					pg_log_error("need at least %d open files, but system limit is %ld",
-								 nclients + 3, (long) rlim.rlim_cur);
+								 nclients + 3, (long) rlim.rlim_max);
 					pg_log_error_hint("Reduce number of clients, or use limit/ulimit to increase the system limit.");
 					exit(1);
 				}
+
+				if (rlim.rlim_cur < nclients + 3)
+				{
+					rlim.rlim_cur = nclients + 3;
+					if (setrlimit(RLIMIT_NOFILE, &rlim) == -1)
+					{
+						pg_log_error("need at least %d open files, but couldn't raise the limit",
+									 nclients + 3);
+						pg_log_error_hint("Reduce number of clients, or use limit/ulimit to increase the system limit.");
+						exit(1);
+					}
+				}
 #endif							/* HAVE_GETRLIMIT */
 				break;
 			case 'C':

base-commit: c407d5426b877b41be87f9e679e321bb2c42e47d
-- 
2.43.0

From ec72e05e87c73dee3de73f9a6586e8e8db2d919e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v4 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 28 +-----------
 src/backend/archive/shell_archive.c      |  6 +--
 src/backend/postmaster/startup.c         | 54 +++++++++++++++++-------
 src/include/postmaster/startup.h         |  3 +-
 4 files changed, 43 insertions(+), 48 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..59a0f191339 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	rc = System(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +305,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = System(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..7977e5a5092 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "postmaster/startup.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -75,10 +76,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = System(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 88eab3d0baf..b0cd40c081e 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
@@ -264,24 +265,45 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
 	proc_exit(0);
 }
 
-void
-PreRestoreCommand(void)
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ */
+int
+System(const char *command, uint32 wait_event_info)
 {
-	/*
-	 * Set in_restore_command to tell the signal handler that we should exit
-	 * right away on SIGTERM. We know that we're at a safe point to do that.
-	 * Check if we had already received the signal, so that we don't miss a
-	 * shutdown request received just before this.
-	 */
-	in_restore_command = true;
-	if (shutdown_requested)
-		proc_exit(1);
-}
+	int			rc;
 
-void
-PostRestoreCommand(void)
-{
-	in_restore_command = false;
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * Set in_restore_command to tell the signal handler that we should
+		 * exit right away on SIGTERM. This is done for the duration of the
+		 * system() call because there isn't a good way to break out while it
+		 * is executing.  Since we might call proc_exit() in a signal handler,
+		 * it is best to put any additional logic outside of this section
+		 * where in_restore_command is set to true.
+		 */
+		in_restore_command = true;
+
+		/*
+		 * Also check if we had already received the signal, so that we don't
+		 * miss a shutdown request received just before this.
+		 */
+		if (shutdown_requested)
+			proc_exit(1);
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		in_restore_command = false;
+
+	pgstat_report_wait_end();
+	return rc;
 }
 
 bool
diff --git a/src/include/postmaster/startup.h b/src/include/postmaster/startup.h
index ae0f6347fc0..09f5dcd3d03 100644
--- a/src/include/postmaster/startup.h
+++ b/src/include/postmaster/startup.h
@@ -27,8 +27,7 @@ extern PGDLLIMPORT int log_startup_progress_interval;
 
 extern void HandleStartupProcInterrupts(void);
 extern void StartupProcessMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
-extern void PreRestoreCommand(void);
-extern void PostRestoreCommand(void);
+extern int	System(const char *command, uint32 wait_event_info);
 extern bool IsPromoteSignaled(void);
 extern void ResetPromoteSignaled(void);
 

base-commit: 454c182f8542890d0e2eac85f70d9a254a34fce3
-- 
2.43.0

From ab826c9c4325860b11e02043c3d77ce015bdbc19 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v4 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml         |  17 ++-
 src/backend/postmaster/startup.c |   3 +
 src/backend/storage/file/fd.c    | 208 +++++++++++++++++++++++++++----
 src/include/storage/fd.h         |   2 +
 4 files changed, 199 insertions(+), 31 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 007746a4429..c69522be67a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2378,15 +2378,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index b0cd40c081e..d3a5e2590ab 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -30,6 +30,7 @@
 #include "storage/pmsignal.h"
 #include "storage/procsignal.h"
 #include "storage/standby.h"
+#include "storage/fd.h"
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
@@ -274,6 +275,7 @@ System(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -303,6 +305,7 @@ System(const char *command, uint32 wait_event_info)
 		in_restore_command = false;
 
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 	return rc;
 }
 
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..9881c18c983 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +952,154 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#else
+	return false;
+#endif
+}
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -968,38 +1123,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1209,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -2724,6 +2875,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call RestoreOriginalOpenFileLimit here,
+	 * but since popen() also opens a file in the current process (this side
+	 * of the pipe) we cannot do so safely. Because we might already have many
+	 * more files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen()
+	 * that calls RestoreOriginalOpenFileLimit only around the actual fork
+	 * call, but that seems too much effort to handle the corner case where
+	 * this external command uses both select() and tries to open more files
+	 * than select() allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From f8a115dd8f32ccd33766caca0b982ea001f1eea1 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v4 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 9881c18c983..a045dc71d2c 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1200,6 +1200,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1214,6 +1215,12 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From ec72e05e87c73dee3de73f9a6586e8e8db2d919e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v5 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 28 +-----------
 src/backend/archive/shell_archive.c      |  6 +--
 src/backend/postmaster/startup.c         | 54 +++++++++++++++++-------
 src/include/postmaster/startup.h         |  3 +-
 4 files changed, 43 insertions(+), 48 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..59a0f191339 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	rc = System(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +305,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = System(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..7977e5a5092 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "postmaster/startup.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -75,10 +76,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = System(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 88eab3d0baf..b0cd40c081e 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
@@ -264,24 +265,45 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
 	proc_exit(0);
 }
 
-void
-PreRestoreCommand(void)
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ */
+int
+System(const char *command, uint32 wait_event_info)
 {
-	/*
-	 * Set in_restore_command to tell the signal handler that we should exit
-	 * right away on SIGTERM. We know that we're at a safe point to do that.
-	 * Check if we had already received the signal, so that we don't miss a
-	 * shutdown request received just before this.
-	 */
-	in_restore_command = true;
-	if (shutdown_requested)
-		proc_exit(1);
-}
+	int			rc;
 
-void
-PostRestoreCommand(void)
-{
-	in_restore_command = false;
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * Set in_restore_command to tell the signal handler that we should
+		 * exit right away on SIGTERM. This is done for the duration of the
+		 * system() call because there isn't a good way to break out while it
+		 * is executing.  Since we might call proc_exit() in a signal handler,
+		 * it is best to put any additional logic outside of this section
+		 * where in_restore_command is set to true.
+		 */
+		in_restore_command = true;
+
+		/*
+		 * Also check if we had already received the signal, so that we don't
+		 * miss a shutdown request received just before this.
+		 */
+		if (shutdown_requested)
+			proc_exit(1);
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		in_restore_command = false;
+
+	pgstat_report_wait_end();
+	return rc;
 }
 
 bool
diff --git a/src/include/postmaster/startup.h b/src/include/postmaster/startup.h
index ae0f6347fc0..09f5dcd3d03 100644
--- a/src/include/postmaster/startup.h
+++ b/src/include/postmaster/startup.h
@@ -27,8 +27,7 @@ extern PGDLLIMPORT int log_startup_progress_interval;
 
 extern void HandleStartupProcInterrupts(void);
 extern void StartupProcessMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
-extern void PreRestoreCommand(void);
-extern void PostRestoreCommand(void);
+extern int	System(const char *command, uint32 wait_event_info);
 extern bool IsPromoteSignaled(void);
 extern void ResetPromoteSignaled(void);
 

base-commit: 454c182f8542890d0e2eac85f70d9a254a34fce3
-- 
2.43.0

From d04d0350510efc57dd1cd5947ef90f139972d3fc Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v5 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml         |  17 ++-
 src/backend/postmaster/startup.c |   3 +
 src/backend/storage/file/fd.c    | 208 +++++++++++++++++++++++++++----
 src/include/storage/fd.h         |   2 +
 4 files changed, 199 insertions(+), 31 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 007746a4429..c69522be67a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2378,15 +2378,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index b0cd40c081e..d3a5e2590ab 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -30,6 +30,7 @@
 #include "storage/pmsignal.h"
 #include "storage/procsignal.h"
 #include "storage/standby.h"
+#include "storage/fd.h"
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
@@ -274,6 +275,7 @@ System(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -303,6 +305,7 @@ System(const char *command, uint32 wait_event_info)
 		in_restore_command = false;
 
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 	return rc;
 }
 
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..f665422692d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +952,154 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#else
+	return;
+#endif
+}
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -968,38 +1123,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1209,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -2724,6 +2875,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call RestoreOriginalOpenFileLimit here,
+	 * but since popen() also opens a file in the current process (this side
+	 * of the pipe) we cannot do so safely. Because we might already have many
+	 * more files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen()
+	 * that calls RestoreOriginalOpenFileLimit only around the actual fork
+	 * call, but that seems too much effort to handle the corner case where
+	 * this external command uses both select() and tries to open more files
+	 * than select() allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From ff352f09fa72d6e359be6bf20a215c3724be2303 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v5 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index f665422692d..03a16e8e5b6 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1200,6 +1200,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1214,6 +1215,12 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From ec72e05e87c73dee3de73f9a6586e8e8db2d919e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v6 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 28 +-----------
 src/backend/archive/shell_archive.c      |  6 +--
 src/backend/postmaster/startup.c         | 54 +++++++++++++++++-------
 src/include/postmaster/startup.h         |  3 +-
 4 files changed, 43 insertions(+), 48 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..59a0f191339 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	rc = System(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +305,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = System(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..7977e5a5092 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "postmaster/startup.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -75,10 +76,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = System(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 88eab3d0baf..b0cd40c081e 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
@@ -264,24 +265,45 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
 	proc_exit(0);
 }
 
-void
-PreRestoreCommand(void)
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ */
+int
+System(const char *command, uint32 wait_event_info)
 {
-	/*
-	 * Set in_restore_command to tell the signal handler that we should exit
-	 * right away on SIGTERM. We know that we're at a safe point to do that.
-	 * Check if we had already received the signal, so that we don't miss a
-	 * shutdown request received just before this.
-	 */
-	in_restore_command = true;
-	if (shutdown_requested)
-		proc_exit(1);
-}
+	int			rc;
 
-void
-PostRestoreCommand(void)
-{
-	in_restore_command = false;
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * Set in_restore_command to tell the signal handler that we should
+		 * exit right away on SIGTERM. This is done for the duration of the
+		 * system() call because there isn't a good way to break out while it
+		 * is executing.  Since we might call proc_exit() in a signal handler,
+		 * it is best to put any additional logic outside of this section
+		 * where in_restore_command is set to true.
+		 */
+		in_restore_command = true;
+
+		/*
+		 * Also check if we had already received the signal, so that we don't
+		 * miss a shutdown request received just before this.
+		 */
+		if (shutdown_requested)
+			proc_exit(1);
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		in_restore_command = false;
+
+	pgstat_report_wait_end();
+	return rc;
 }
 
 bool
diff --git a/src/include/postmaster/startup.h b/src/include/postmaster/startup.h
index ae0f6347fc0..09f5dcd3d03 100644
--- a/src/include/postmaster/startup.h
+++ b/src/include/postmaster/startup.h
@@ -27,8 +27,7 @@ extern PGDLLIMPORT int log_startup_progress_interval;
 
 extern void HandleStartupProcInterrupts(void);
 extern void StartupProcessMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
-extern void PreRestoreCommand(void);
-extern void PostRestoreCommand(void);
+extern int	System(const char *command, uint32 wait_event_info);
 extern bool IsPromoteSignaled(void);
 extern void ResetPromoteSignaled(void);
 

base-commit: 454c182f8542890d0e2eac85f70d9a254a34fce3
-- 
2.43.0

From d03ad921f8bca8c16107d755d6edc75d1ccba3e0 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v6 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml         |  17 ++-
 src/backend/postmaster/startup.c |   3 +
 src/backend/storage/file/fd.c    | 206 +++++++++++++++++++++++++++----
 src/include/storage/fd.h         |   2 +
 4 files changed, 197 insertions(+), 31 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 007746a4429..c69522be67a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2378,15 +2378,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index b0cd40c081e..d3a5e2590ab 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -30,6 +30,7 @@
 #include "storage/pmsignal.h"
 #include "storage/procsignal.h"
 #include "storage/standby.h"
+#include "storage/fd.h"
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
@@ -274,6 +275,7 @@ System(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -303,6 +305,7 @@ System(const char *command, uint32 wait_event_info)
 		in_restore_command = false;
 
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 	return rc;
 }
 
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e454db4c020..fc60e7ebeb9 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +952,152 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#endif
+}
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -968,38 +1121,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1207,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -2724,6 +2873,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call RestoreOriginalOpenFileLimit here,
+	 * but since popen() also opens a file in the current process (this side
+	 * of the pipe) we cannot do so safely. Because we might already have many
+	 * more files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen()
+	 * that calls RestoreOriginalOpenFileLimit only around the actual fork
+	 * call, but that seems too much effort to handle the corner case where
+	 * this external command uses both select() and tries to open more files
+	 * than select() allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From faf1c3b550215a6dc831d5fc9ee4e5c2ec5604a4 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v6 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index fc60e7ebeb9..f0643838857 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1198,6 +1198,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1212,6 +1213,12 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From 249ebbac1b6c01b99795cfe9b0201ab7ee6bacfa Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v7 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 28 +-----------
 src/backend/archive/shell_archive.c      |  6 +--
 src/backend/postmaster/startup.c         | 54 +++++++++++++++++-------
 src/include/postmaster/startup.h         |  3 +-
 4 files changed, 43 insertions(+), 48 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..59a0f191339 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,7 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	rc = System(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +305,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = System(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..7977e5a5092 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "postmaster/startup.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -75,10 +76,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = System(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..2e3d0ea8cf0 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
@@ -264,24 +265,45 @@ StartupProcessMain(const void *startup_data, size_t startup_data_len)
 	proc_exit(0);
 }
 
-void
-PreRestoreCommand(void)
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ */
+int
+System(const char *command, uint32 wait_event_info)
 {
-	/*
-	 * Set in_restore_command to tell the signal handler that we should exit
-	 * right away on SIGTERM. We know that we're at a safe point to do that.
-	 * Check if we had already received the signal, so that we don't miss a
-	 * shutdown request received just before this.
-	 */
-	in_restore_command = true;
-	if (shutdown_requested)
-		proc_exit(1);
-}
+	int			rc;
 
-void
-PostRestoreCommand(void)
-{
-	in_restore_command = false;
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * Set in_restore_command to tell the signal handler that we should
+		 * exit right away on SIGTERM. This is done for the duration of the
+		 * system() call because there isn't a good way to break out while it
+		 * is executing.  Since we might call proc_exit() in a signal handler,
+		 * it is best to put any additional logic outside of this section
+		 * where in_restore_command is set to true.
+		 */
+		in_restore_command = true;
+
+		/*
+		 * Also check if we had already received the signal, so that we don't
+		 * miss a shutdown request received just before this.
+		 */
+		if (shutdown_requested)
+			proc_exit(1);
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		in_restore_command = false;
+
+	pgstat_report_wait_end();
+	return rc;
 }
 
 bool
diff --git a/src/include/postmaster/startup.h b/src/include/postmaster/startup.h
index ec2c8d3bff5..7fdf9100044 100644
--- a/src/include/postmaster/startup.h
+++ b/src/include/postmaster/startup.h
@@ -27,8 +27,7 @@ extern PGDLLIMPORT int log_startup_progress_interval;
 
 extern void ProcessStartupProcInterrupts(void);
 pg_noreturn extern void StartupProcessMain(const void *startup_data, size_t startup_data_len);
-extern void PreRestoreCommand(void);
-extern void PostRestoreCommand(void);
+extern int	System(const char *command, uint32 wait_event_info);
 extern bool IsPromoteSignaled(void);
 extern void ResetPromoteSignaled(void);
 

base-commit: 65db3963ae7154b8f01e4d73dc6b1ffd81c70e1e
-- 
2.43.0

From 5e861b43cc0dc903d530e9140ddf5acf5c81b270 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v7 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 doc/src/sgml/config.sgml         |  17 ++-
 src/backend/postmaster/startup.c |   3 +
 src/backend/storage/file/fd.c    | 206 +++++++++++++++++++++++++++----
 src/include/storage/fd.h         |   2 +
 4 files changed, 197 insertions(+), 31 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 3d62c8bd274..271ae897172 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2378,15 +2378,14 @@ include_dir 'conf.d'
       </term>
       <listitem>
        <para>
-        Sets the maximum number of simultaneously open files allowed to each
-        server subprocess. The default is one thousand files. If the kernel is enforcing
-        a safe per-process limit, you don't need to worry about this setting.
-        But on some platforms (notably, most BSD systems), the kernel will
-        allow individual processes to open many more files than the system
-        can actually support if many processes all try to open
-        that many files. If you find yourself seeing <quote>Too many open
-        files</quote> failures, try reducing this setting.
-        This parameter can only be set at server start.
+        Sets the maximum number of files to each server subprocess is allowed
+        to open. The default is one thousand files. If the kernel is enforcing
+        a lower soft limit Postgres will automatically increase it if the hard
+        limit allows that. Setting this value too high can cause resource
+        exhaustion problems: On some platforms (notably, most BSD systems), the
+        kernel will allow individual processes to open many more files than the
+        system can actually support if many processes all try to open that many
+        files. This parameter can only be set at server start.
        </para>
       </listitem>
      </varlistentry>
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 2e3d0ea8cf0..9a5c3eb29a7 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -30,6 +30,7 @@
 #include "storage/pmsignal.h"
 #include "storage/procsignal.h"
 #include "storage/standby.h"
+#include "storage/fd.h"
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
@@ -274,6 +275,7 @@ System(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	RestoreOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -303,6 +305,7 @@ System(const char *command, uint32 wait_event_info)
 		in_restore_command = false;
 
 	pgstat_report_wait_end();
+	RestoreCustomOpenFileLimit();
 	return rc;
 }
 
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 62f1185859f..de6bdf9c141 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -157,6 +157,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -945,6 +952,152 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#endif
+}
+
+/*
+ * RestoreOriginalOpenFileLimit --- Restore the original open file limit that
+ * 		was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+void
+RestoreOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * RestoreCustomOpenFileLimit --- Restores our custom open file limit after a
+ * 		previous call to RestoreOriginalOpenFileLimit.
+ */
+void
+RestoreCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -968,38 +1121,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -1053,15 +1207,10 @@ set_max_safe_fds(void)
 	 * or the experimentally-determined EMFILE limit.
 	 *----------
 	 */
-	count_usable_fds(max_files_per_process,
+	count_usable_fds(max_files_per_process + NUM_RESERVED_FDS,
 					 &usable_fds, &already_open);
 
-	max_safe_fds = Min(usable_fds, max_files_per_process - already_open);
-
-	/*
-	 * Take off the FDs reserved for system() etc.
-	 */
-	max_safe_fds -= NUM_RESERVED_FDS;
+	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
 	/*
 	 * Make sure we still have enough to get by.
@@ -2724,6 +2873,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call RestoreOriginalOpenFileLimit here,
+	 * but since popen() also opens a file in the current process (this side
+	 * of the pipe) we cannot do so safely. Because we might already have many
+	 * more files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen()
+	 * that calls RestoreOriginalOpenFileLimit only around the actual fork
+	 * call, but that seems too much effort to handle the corner case where
+	 * this external command uses both select() and tries to open more files
+	 * than select() allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index e3067ab6597..d24e7f1c8df 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -177,6 +177,8 @@ extern void RemovePgTempFiles(void);
 extern void RemovePgTempFilesInDir(const char *tmpdirname, bool missing_ok,
 								   bool unlink_all);
 extern bool looks_like_temp_rel_name(const char *name);
+extern void RestoreOriginalOpenFileLimit(void);
+extern void RestoreCustomOpenFileLimit(void);
 
 extern int	pg_fsync(int fd);
 extern int	pg_fsync_no_writethrough(int fd);
-- 
2.43.0

From af312866227074205e96fb80ccf86d74ff050d71 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v7 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index de6bdf9c141..1454568810d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1198,6 +1198,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1212,6 +1213,12 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds - NUM_RESERVED_FDS, max_files_per_process);
 
+	/* Update GUC variable to allow users to see the result */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Make sure we still have enough to get by.
 	 */
-- 
2.43.0

From 5dbab2ccf0d74313dbc2cbaeb418346de8cc2f48 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v8 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 29 ++--------------
 src/backend/archive/shell_archive.c      |  5 +--
 src/backend/postmaster/startup.c         |  1 +
 src/backend/storage/file/fd.c            | 42 ++++++++++++++++++++++++
 src/include/storage/fd.h                 |  1 +
 5 files changed, 48 insertions(+), 30 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..c7640ec5025 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,8 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	/* Copy xlog from archival storage to XLOGDIR */
+	rc = pg_system(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +306,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = pg_system(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..64c2c6aa760 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -75,10 +75,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = pg_system(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 7149a67fcbc..eb79fda1fd9 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 0e8299dd556..718d8079ad7 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2734,6 +2734,48 @@ OpenTransientFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
 	return -1;					/* failure */
 }
 
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ *
+ * If WAIT_EVENT_RESTORE_COMMAND is passed as the wait_event_info, it will also
+ * call the necessary PreRestoreCommand/PostRerstoreCommand functions. It's
+ * unfortunate that we have to do couple the behaviour of this function so
+ * tighlty to the restore command logic, but it's the only way to make sure
+ * that we don't have additional logic inbetween the PreRestoreCommand and
+ * PostRestoreCommand calls.
+ */
+int
+pg_system(const char *command, uint32 wait_event_info)
+{
+	int			rc;
+
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * PreRestoreCommand() informs the SIGTERM handler for the startup
+		 * process that it should proc_exit() right away.  This is done for
+		 * the duration of the system() call because there isn't a good way to
+		 * break out while it is executing.  Since we might call proc_exit()
+		 * in a signal handler, it is best to put any additional logic before
+		 * or after the PreRestoreCommand()/PostRestoreCommand() section.
+		 */
+		PreRestoreCommand();
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		PostRestoreCommand();
+
+	pgstat_report_wait_end();
+	return rc;
+}
+
+
 /*
  * Routines that want to initiate a pipe stream should use OpenPipeStream
  * rather than plain popen().  This lets fd.c deal with freeing FDs if
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2d445674a1a 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -187,6 +187,7 @@ extern int	pg_fsync_writethrough(int fd);
 extern int	pg_fdatasync(int fd);
 extern bool pg_file_exists(const char *name);
 extern void pg_flush_data(int fd, off_t offset, off_t nbytes);
+extern int	pg_system(const char *command, uint32 wait_event_info);
 extern int	pg_truncate(const char *path, off_t length);
 extern void fsync_fname(const char *fname, bool isdir);
 extern int	fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel);

base-commit: f09088a01d3372fdfe5da12dd0b2e24989f0caa6
-- 
2.43.0

From c43950afa51b22892211770a41d4be4609b7b9ac Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v8 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 src/backend/storage/file/fd.c | 199 +++++++++++++++++++++++++++++++---
 1 file changed, 184 insertions(+), 15 deletions(-)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 718d8079ad7..b3a58deef43 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -158,6 +158,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -946,6 +953,152 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#endif
+}
+
+/*
+ * UseOriginalOpenFileLimit --- Makes the process use the original open file
+ * 		limit that was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+static void
+UseOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * UseCustomOpenFileLimit --- Makes the process use our custom open file limit
+ * 		after that we configured based on the max_files_per_process GUC.
+ */
+static void
+UseCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -969,38 +1122,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -2750,6 +2904,7 @@ pg_system(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	UseOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -2772,6 +2927,7 @@ pg_system(const char *command, uint32 wait_event_info)
 		PostRestoreCommand();
 
 	pgstat_report_wait_end();
+	UseCustomOpenFileLimit();
 	return rc;
 }
 
@@ -2805,6 +2961,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call UseOriginalOpenFileLimit here, but
+	 * since popen() also opens a file in the current process (this side of the
+	 * pipe) we cannot do so safely. Because we might already have many more
+	 * files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen() that
+	 * calls UseOriginalOpenFileLimit only around the actual fork call, but
+	 * that seems too much effort to handle the corner case where this external
+	 * command uses both select() and tries to open more files than select()
+	 * allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
-- 
2.43.0

From 298c3c436eb4535df95d7efb0b17105cc6e6c770 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v8 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index b3a58deef43..8cee38e6c17 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1199,6 +1199,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1214,6 +1215,16 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds, max_files_per_process);
 
+	/*
+	 * Update GUC variable to allow users to see if the result is different
+	 * than what the used value turns out to be different than what they had
+	 * configured.
+	 */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Take off the FDs reserved for system() etc.
 	 */
-- 
2.43.0

From 9e478de009b5c1a77ca47fc268cb51adbb657dd8 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Sun, 23 Feb 2025 16:52:29 +0100
Subject: [PATCH v8 1/3] Adds a helper for places that shell out to system()

We need to call system() in a few places, and to do so safely we need
some pre/post-fork logic. This encapsulates that logic into a single
System helper function. The main reason to do this is that in a follow
up commit we want to add even more logic pre and post-fork.
---
 src/backend/access/transam/xlogarchive.c | 29 ++--------------
 src/backend/archive/shell_archive.c      |  6 ++--
 src/backend/postmaster/startup.c         |  1 +
 src/backend/storage/file/fd.c            | 42 ++++++++++++++++++++++++
 src/include/storage/fd.h                 |  1 +
 5 files changed, 49 insertions(+), 30 deletions(-)

diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..c7640ec5025 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,27 +158,8 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
-	/*
-	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
-	 * that it should proc_exit() right away.  This is done for the duration
-	 * of the system() call because there isn't a good way to break out while
-	 * it is executing.  Since we might call proc_exit() in a signal handler,
-	 * it is best to put any additional logic before or after the
-	 * PreRestoreCommand()/PostRestoreCommand() section.
-	 */
-	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
-	PostRestoreCommand();
-
-	pgstat_report_wait_end();
+	/* Copy xlog from archival storage to XLOGDIR */
+	rc = pg_system(xlogRestoreCmd, WAIT_EVENT_RESTORE_COMMAND);
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -325,11 +306,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	/*
 	 * execute the constructed command
 	 */
-	fflush(NULL);
-	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
-	pgstat_report_wait_end();
-
+	rc = pg_system(xlogRecoveryCmd, wait_event_info);
 	pfree(xlogRecoveryCmd);
 
 	if (rc != 0)
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..5dd8e2c7247 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "storage/fd.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -75,10 +76,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 			(errmsg_internal("executing archive command \"%s\"",
 							 xlogarchcmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
-	pgstat_report_wait_end();
+	rc = pg_system(xlogarchcmd, WAIT_EVENT_ARCHIVE_COMMAND);
 
 	if (rc != 0)
 	{
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..783eb88e59d 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -33,6 +33,7 @@
 #include "utils/guc.h"
 #include "utils/memutils.h"
 #include "utils/timeout.h"
+#include "utils/wait_event_types.h"
 
 
 #ifndef USE_POSTMASTER_DEATH_SIGNAL
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index a4ec7959f31..9f3b58c3767 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -2731,6 +2731,48 @@ OpenTransientFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
 	return -1;					/* failure */
 }
 
+/*
+ * A custom wrapper around the system() function that calls the necessary
+ * functions pre/post-fork.
+ *
+ * If WAIT_EVENT_RESTORE_COMMAND is passed as the wait_event_info, it will also
+ * call the necessary PreRestoreCommand/PostRerstoreCommand functions. It's
+ * unfortunate that we have to do couple the behaviour of this function so
+ * tighlty to the restore command logic, but it's the only way to make sure
+ * that we don't have additional logic inbetween the PreRestoreCommand and
+ * PostRestoreCommand calls.
+ */
+int
+pg_system(const char *command, uint32 wait_event_info)
+{
+	int			rc;
+
+	fflush(NULL);
+	pgstat_report_wait_start(wait_event_info);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+	{
+		/*
+		 * PreRestoreCommand() informs the SIGTERM handler for the startup
+		 * process that it should proc_exit() right away.  This is done for
+		 * the duration of the system() call because there isn't a good way to
+		 * break out while it is executing.  Since we might call proc_exit()
+		 * in a signal handler, it is best to put any additional logic before
+		 * or after the PreRestoreCommand()/PostRestoreCommand() section.
+		 */
+		PreRestoreCommand();
+	}
+
+	rc = system(command);
+
+	if (wait_event_info == WAIT_EVENT_RESTORE_COMMAND)
+		PostRestoreCommand();
+
+	pgstat_report_wait_end();
+	return rc;
+}
+
+
 /*
  * Routines that want to initiate a pipe stream should use OpenPipeStream
  * rather than plain popen().  This lets fd.c deal with freeing FDs if
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index b77d8e5e30e..2d445674a1a 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -187,6 +187,7 @@ extern int	pg_fsync_writethrough(int fd);
 extern int	pg_fdatasync(int fd);
 extern bool pg_file_exists(const char *name);
 extern void pg_flush_data(int fd, off_t offset, off_t nbytes);
+extern int	pg_system(const char *command, uint32 wait_event_info);
 extern int	pg_truncate(const char *path, off_t length);
 extern void fsync_fname(const char *fname, bool isdir);
 extern int	fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel);

base-commit: ad25744f436ed7809fc754e1a44630b087812fbc
-- 
2.51.1

From 557a63fb212e89f0448b08025fb5c88c46ea198e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Tue, 11 Feb 2025 19:15:36 +0100
Subject: [PATCH v8 2/3] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work that Andres is doing, because
io_uring consumes a lot of file descriptors. Even without looking at AIO
there is a large number of reports from people that require changing
their soft file limit before starting Postgres, sometimes falling back
to lowering max_files_per_process when they fail to do so[3-8]. It's
also not all that strange to fail at setting the soft open file limit
because there are multiple places where one can configure such limits
and usually only one of them is effective (which one depends on how
Postgres is started). In cloud environments its also often not possible
for user to change the soft limit, because they don't control the way
that Postgres is started.

One thing to note is that we temporarily restore the original soft
limit when shell-ing out to other executables. This is done as a
precaution in case those executables are using select(2).

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 src/backend/storage/file/fd.c | 199 +++++++++++++++++++++++++++++++---
 1 file changed, 184 insertions(+), 15 deletions(-)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 9f3b58c3767..37de12fbb7e 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -158,6 +158,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -943,6 +950,152 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#endif
+}
+
+/*
+ * UseOriginalOpenFileLimit --- Makes the process use the original open file
+ * 		limit that was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+static void
+UseOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
+/*
+ * UseCustomOpenFileLimit --- Makes the process use our custom open file limit
+ * 		after that we configured based on the max_files_per_process GUC.
+ */
+static void
+UseCustomOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &custom_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -966,38 +1119,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -2747,6 +2901,7 @@ pg_system(const char *command, uint32 wait_event_info)
 {
 	int			rc;
 
+	UseOriginalOpenFileLimit();
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
 
@@ -2769,6 +2924,7 @@ pg_system(const char *command, uint32 wait_event_info)
 		PostRestoreCommand();
 
 	pgstat_report_wait_end();
+	UseCustomOpenFileLimit();
 	return rc;
 }
 
@@ -2802,6 +2958,19 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
+
+	/*
+	 * It would be great if we could call UseOriginalOpenFileLimit here, but
+	 * since popen() also opens a file in the current process (this side of the
+	 * pipe) we cannot do so safely. Because we might already have many more
+	 * files open than the original limit.
+	 *
+	 * The only way to address this would be implementing a custom popen() that
+	 * calls UseOriginalOpenFileLimit only around the actual fork call, but
+	 * that seems too much effort to handle the corner case where this external
+	 * command uses both select() and tries to open more files than select()
+	 * allows for.
+	 */
 	fflush(NULL);
 	pqsignal(SIGPIPE, SIG_DFL);
 	errno = 0;
-- 
2.51.1

From 6bf857b0ddb43b2a5abb62bcdc5669debc96edab Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <github-tech@jeltef.nl>
Date: Wed, 12 Feb 2025 01:08:07 +0100
Subject: [PATCH v8 3/3] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 37de12fbb7e..3cd99054c9f 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1196,6 +1196,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1211,6 +1212,16 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds, max_files_per_process);
 
+	/*
+	 * Update GUC variable to allow users to see if the result is different
+	 * than what the used value turns out to be different than what they had
+	 * configured.
+	 */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Take off the FDs reserved for system() etc.
 	 */
-- 
2.51.1

From 699c49d1f75c143277007b6171c9499bfb6757fb Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Sun, 7 Dec 2025 15:07:08 +0100
Subject: [PATCH v9 1/2] Bump postmaster soft open file limit (RLIMIT_NOFILE)
 when necessary

The default open file limit of 1024 on Linux is extremely low. The
reason that this hasn't changed change is because doing so would break
legacy programs that use the select(2) system call in hard to debug
ways. So instead programs that want to opt-in to a higher open file
limit are expected to bump their soft limit to their hard limit on
startup. Details on this are very well explained in a blogpost by the
systemd author[1]. There's also a similar change done by the Go
language[2].

This starts bumping postmaster its soft open file limit when we realize
that we'll run into the soft limit with the requested
max_files_per_process GUC. We do so by slightly changing the meaning of
the max_files_per_process GUC. The actual (not publicly exposed) limit
is max_safe_fds, previously this would be set to:
max_files_per_process - already_open_files - NUM_RESERVED_FDS
After this change we now try to set max_safe_fds to
max_files_per_process if the system allows that. This is deemed more
natural to understand for users, because now the limit of files that
they can open is actually what they configured in max_files_per_process.

Adding this infrastructure to change RLIMIT_NOFILE when needed is
especially useful for the AIO work, because io_uring consumes a lot of
file descriptors. Even without looking at AIO there is a large number of
reports from people that require changing their soft file limit before
starting Postgres, sometimes falling back to lowering
max_files_per_process when they fail to do so[3-8].
It's also not all that strange to fail at setting the soft open file
limit because there are multiple places where one can configure such
limits and usually only one of them is effective (which one depends on
how Postgres is started). In cloud environments its also often not
possible for user to change the soft limit, because they don't control
the way that Postgres is started. Finally, for the multi-threading work
this is also very important because then the open file limit is not
per-backend anymore, but for the whole system.

The most complex change in this patch is how we shell out to other
systems executables. Instead of using system and popen directly we now
implement our own versions of these commands that restore the original
file limits before starting the other executables. This is done as a
precaution in case those executables are using select(2).

P.S. The initdb test needed to be updated because on our OpenBSD CI the
stderr would now contain:

LOG:  increased open file limit to 1004

[1]: https://0pointer.net/blog/file-descriptor-limits.html
[2]: https://github.com/golang/go/issues/46279
[3]: https://serverfault.com/questions/785330/getting-too-many-open-files-error-for-postgres
[4]: https://serverfault.com/questions/716982/how-to-raise-max-no-of-file-descriptors-for-daemons-running-on-debian-jessie
[5]: https://www.postgresql.org/message-id/flat/CAKtc8vXh7NvP_qWj8EqqorPY97bvxSaX3h5u7a9PptRFHW5x7g%40mail.gmail.com
[6]: https://www.postgresql.org/message-id/flat/113ce31b0908120955w77029099i7ececc053084095a%40mail.gmail.com
[7]: https://github.com/abiosoft/colima/discussions/836
[8]: https://www.postgresql.org/message-id/flat/29663.1007738957%40sss.pgh.pa.us#2079ec9e2d8b251593812a3711bfe9e9
---
 meson.build                              |   1 +
 src/backend/access/transam/xlogarchive.c |  18 +-
 src/backend/archive/shell_archive.c      |   3 +-
 src/backend/storage/file/fd.c            | 442 +++++++++++++++++++++--
 src/bin/initdb/t/001_initdb.pl           |   6 +-
 src/include/pg_config.h.in               |   3 +
 src/include/storage/fd.h                 |   1 +
 7 files changed, 437 insertions(+), 37 deletions(-)

diff --git a/meson.build b/meson.build
index 6e7ddd74683..ac778b87818 100644
--- a/meson.build
+++ b/meson.build
@@ -2891,6 +2891,7 @@ func_checks = [
   ['localeconv_l'],
   ['mbstowcs_l'],
   ['mkdtemp'],
+  ['pipe2'],
   ['posix_fadvise'],
   ['posix_fallocate'],
   ['ppoll'],
diff --git a/src/backend/access/transam/xlogarchive.c b/src/backend/access/transam/xlogarchive.c
index 1ef1713c91a..5a99ddc291e 100644
--- a/src/backend/access/transam/xlogarchive.c
+++ b/src/backend/access/transam/xlogarchive.c
@@ -158,10 +158,9 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 			(errmsg_internal("executing restore command \"%s\"",
 							 xlogRestoreCmd)));
 
-	fflush(NULL);
-	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
-
 	/*
+	 * Copy xlog from archival storage to XLOGDIR
+	 *
 	 * PreRestoreCommand() informs the SIGTERM handler for the startup process
 	 * that it should proc_exit() right away.  This is done for the duration
 	 * of the system() call because there isn't a good way to break out while
@@ -169,16 +168,13 @@ RestoreArchivedFile(char *path, const char *xlogfname,
 	 * it is best to put any additional logic before or after the
 	 * PreRestoreCommand()/PostRestoreCommand() section.
 	 */
+	fflush(NULL);
+	pgstat_report_wait_start(WAIT_EVENT_RESTORE_COMMAND);
 	PreRestoreCommand();
-
-	/*
-	 * Copy xlog from archival storage to XLOGDIR
-	 */
-	rc = system(xlogRestoreCmd);
-
+	rc = pg_system(xlogRestoreCmd);
 	PostRestoreCommand();
-
 	pgstat_report_wait_end();
+
 	pfree(xlogRestoreCmd);
 
 	if (rc == 0)
@@ -327,7 +323,7 @@ ExecuteRecoveryCommand(const char *command, const char *commandName,
 	 */
 	fflush(NULL);
 	pgstat_report_wait_start(wait_event_info);
-	rc = system(xlogRecoveryCmd);
+	rc = pg_system(xlogRecoveryCmd);
 	pgstat_report_wait_end();
 
 	pfree(xlogRecoveryCmd);
diff --git a/src/backend/archive/shell_archive.c b/src/backend/archive/shell_archive.c
index 828723afe47..beb81ee7d9d 100644
--- a/src/backend/archive/shell_archive.c
+++ b/src/backend/archive/shell_archive.c
@@ -22,6 +22,7 @@
 #include "archive/shell_archive.h"
 #include "common/percentrepl.h"
 #include "pgstat.h"
+#include "storage/fd.h"
 
 static bool shell_archive_configured(ArchiveModuleState *state);
 static bool shell_archive_file(ArchiveModuleState *state,
@@ -77,7 +78,7 @@ shell_archive_file(ArchiveModuleState *state, const char *file,
 
 	fflush(NULL);
 	pgstat_report_wait_start(WAIT_EVENT_ARCHIVE_COMMAND);
-	rc = system(xlogarchcmd);
+	rc = pg_system(xlogarchcmd);
 	pgstat_report_wait_end();
 
 	if (rc != 0)
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index e9eaaf9c829..40212d733e9 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -80,6 +80,7 @@
 #include <sys/types.h>
 #ifndef WIN32
 #include <sys/mman.h>
+#include <sys/wait.h>
 #endif
 #include <limits.h>
 #include <unistd.h>
@@ -91,6 +92,7 @@
 #include "common/file_perm.h"
 #include "common/file_utils.h"
 #include "common/pg_prng.h"
+#include "libpq/pqsignal.h"
 #include "miscadmin.h"
 #include "pgstat.h"
 #include "postmaster/startup.h"
@@ -158,6 +160,13 @@ int			max_files_per_process = 1000;
  */
 int			max_safe_fds = FD_MINFREE;	/* default if not changed */
 
+#ifdef HAVE_GETRLIMIT
+static bool saved_original_max_open_files;
+static struct rlimit original_max_open_files;
+static struct rlimit custom_max_open_files;
+#endif
+
+
 /* Whether it is safe to continue running after fsync() fails. */
 bool		data_sync_retry = false;
 
@@ -261,6 +270,11 @@ typedef struct
 		FILE	   *file;
 		DIR		   *dir;
 		int			fd;
+		struct
+		{
+			FILE	   *file;
+			pid_t		pid;
+		}			pipe;
 	}			desc;
 } AllocateDesc;
 
@@ -943,6 +957,128 @@ InitTemporaryFileAccess(void)
 #endif
 }
 
+/*
+ * Returns true if the passed in highestfd is the last one that we're allowed
+ * to open based on our. This should only be called if
+ */
+static bool
+IsOpenFileLimit(int highestfd)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	return highestfd >= custom_max_open_files.rlim_cur - 1;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Increases the open file limit (RLIMIT_NOFILE) by the requested amount.
+ * Returns true if successful, false otherwise.
+ */
+static bool
+IncreaseOpenFileLimit(int extra_files)
+{
+#ifdef HAVE_GETRLIMIT
+	struct rlimit rlim;
+
+	if (!saved_original_max_open_files)
+	{
+		return false;
+	}
+
+	rlim = custom_max_open_files;
+
+	/* If we're already at the max we reached our limit */
+	if (rlim.rlim_cur == original_max_open_files.rlim_max)
+		return false;
+
+	/* Otherwise try to increase the soft limit to what we need */
+	rlim.rlim_cur = Min(rlim.rlim_cur + extra_files, rlim.rlim_max);
+
+	if (setrlimit(RLIMIT_NOFILE, &rlim) != 0)
+	{
+		/* We made sure not to exceed the hard limit, so this shouldn't fail */
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+		return false;
+	}
+
+	custom_max_open_files = rlim;
+
+	elog(LOG, "increased open file limit to %ld", (long) rlim.rlim_cur);
+
+	return true;
+#else
+	return false;
+#endif
+}
+
+/*
+ * Saves the original open file limit (RLIMIT_NOFILE) the first time when this
+ * is called. If called again it's a no-op.
+ *
+ * Returns true if successful, false otherwise.
+ */
+static void
+SaveOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	int			status;
+
+	if (saved_original_max_open_files)
+	{
+		/* Already saved, no need to do it again */
+		return;
+	}
+
+	status = getrlimit(RLIMIT_NOFILE, &original_max_open_files);
+	if (status != 0)
+	{
+		ereport(WARNING, (errmsg("getrlimit failed: %m")));
+		return;
+	}
+
+	custom_max_open_files = original_max_open_files;
+	saved_original_max_open_files = true;
+	return;
+#endif
+}
+
+#ifndef WIN32
+/*
+ * UseOriginalOpenFileLimit --- Makes the process use the original open file
+ * 		limit that was present at postmaster start.
+ *
+ * This should be called before spawning subprocesses that might use select(2)
+ * which can only handle file descriptors up to 1024.
+ */
+static void
+UseOriginalOpenFileLimit(void)
+{
+#ifdef HAVE_GETRLIMIT
+	if (!saved_original_max_open_files)
+	{
+		return;
+	}
+
+	if (custom_max_open_files.rlim_cur == original_max_open_files.rlim_cur)
+	{
+		/* Not changed, so no need to call setrlimit at all */
+		return;
+	}
+
+	if (setrlimit(RLIMIT_NOFILE, &original_max_open_files) != 0)
+	{
+		ereport(WARNING, (errmsg("setrlimit failed: %m")));
+	}
+#endif
+}
+#endif							/* WIN32 */
+
 /*
  * count_usable_fds --- count how many FDs the system will let us open,
  *		and estimate how many are already open.
@@ -966,38 +1102,39 @@ count_usable_fds(int max_to_probe, int *usable_fds, int *already_open)
 	int			highestfd = 0;
 	int			j;
 
-#ifdef HAVE_GETRLIMIT
-	struct rlimit rlim;
-	int			getrlimit_status;
-#endif
-
 	size = 1024;
 	fd = (int *) palloc(size * sizeof(int));
 
-#ifdef HAVE_GETRLIMIT
-	getrlimit_status = getrlimit(RLIMIT_NOFILE, &rlim);
-	if (getrlimit_status != 0)
-		ereport(WARNING, (errmsg("getrlimit failed: %m")));
-#endif							/* HAVE_GETRLIMIT */
+	SaveOriginalOpenFileLimit();
 
 	/* dup until failure or probe limit reached */
 	for (;;)
 	{
 		int			thisfd;
 
-#ifdef HAVE_GETRLIMIT
-
 		/*
 		 * don't go beyond RLIMIT_NOFILE; causes irritating kernel logs on
 		 * some platforms
 		 */
-		if (getrlimit_status == 0 && highestfd >= rlim.rlim_cur - 1)
-			break;
-#endif
+		if (IsOpenFileLimit(highestfd))
+		{
+			if (!IncreaseOpenFileLimit(max_to_probe - used))
+				break;
+		}
 
 		thisfd = dup(2);
 		if (thisfd < 0)
 		{
+			/*
+			 * Eventhough we do the pre-check above, it's still possible that
+			 * the call to dup fails with EMFILE. This can happen if the last
+			 * file descriptor was already assigned to an "already open" file.
+			 * One example of this happening, is if we're already at the soft
+			 * limit when we call count_usable_fds.
+			 */
+			if (errno == EMFILE && IncreaseOpenFileLimit(max_to_probe - used))
+				continue;
+
 			/* Expect EMFILE or ENFILE, else it's fishy */
 			if (errno != EMFILE && errno != ENFILE)
 				elog(WARNING, "duplicating stderr file descriptor failed after %d successes: %m", used);
@@ -2731,6 +2868,265 @@ OpenTransientFilePerm(const char *fileName, int fileFlags, mode_t fileMode)
 	return -1;					/* failure */
 }
 
+#ifndef WIN32
+/*
+ * Helper function to fork a child process for executing a shell command.
+ *
+ * This handles all the standard child process setup:
+ * - Blocks signals before fork to avoid race conditions
+ * - Lowers the file limit in the child (UseOriginalOpenFileLimit)
+ * - Resets signal handlers that the backend has set to SIG_IGN
+ * - Unblocks signals in both parent and child after setup
+ *
+ * Returns -1 on fork failure, 0 in the child, or the child PID in the parent.
+ * After this returns in the child, the caller should do any additional setup
+ * (like pipe redirection) and then call exec_shell_command().
+ */
+static pid_t
+fork_for_shell_command(void)
+{
+	pid_t		pid;
+	sigset_t	save_mask;
+
+	/* Block signals during fork to avoid race conditions in child setup */
+	sigprocmask(SIG_SETMASK, &BlockSig, &save_mask);
+
+	pid = fork();
+	if (pid == 0)
+	{
+		/* Child process */
+		UseOriginalOpenFileLimit();
+
+		/*
+		 * Reset signal handlers to default. The backend ignores these, and
+		 * SIG_IGN is preserved across exec, so we must reset them to allow
+		 * the child process to respond to signals normally.
+		 */
+		pqsignal(SIGINT, SIG_DFL);
+		pqsignal(SIGQUIT, SIG_DFL);
+		pqsignal(SIGPIPE, SIG_DFL);
+	}
+
+	/* Restore/unblock signals in both parent and child (also if fork failed) */
+	sigprocmask(SIG_SETMASK, &save_mask, NULL);
+
+	return pid;
+}
+
+/*
+ * Execute a shell command via /bin/sh. This function does not return.
+ * Should be called in the child process after fork_for_shell_command().
+ */
+static pg_noreturn void
+exec_shell_command(const char *command)
+{
+	execl("/bin/sh", "sh", "-c", command, (char *) NULL);
+	_exit(127);
+}
+#endif
+
+/*
+ * pg_system - Custom system() that lowers file limit in child process.
+ *
+ * This is needed because we may have bumped the soft RLIMIT_NOFILE limit
+ * above its original value. The child process should use the original limit
+ * to avoid issues with programs that use select(2), which can only handle
+ * FDs up to FD_SETSIZE (typically 1024).
+ *
+ * We can't just call UseOriginalOpenFileLimit() before the standard system()
+ * because that would temporarily lower the limit in the parent process,
+ * which could cause concurrent file operations to fail in a multithreaded
+ * environment.
+ */
+int
+pg_system(const char *command)
+{
+#ifndef WIN32
+	pid_t		pid;
+	int			status;
+	pid_t		waitresult;
+
+	pid = fork_for_shell_command();
+	if (pid == (pid_t) -1)
+		return -1;
+	if (pid == 0)
+		exec_shell_command(command);	/* does not return */
+
+	/* Wait for child */
+	do
+	{
+		waitresult = waitpid(pid, &status, 0);
+	} while (waitresult == (pid_t) -1 && errno == EINTR);
+
+	if (waitresult != pid)
+		return -1;
+
+	return status;
+#else
+	/* On Windows, just use standard system() - no RLIMIT_NOFILE bumping there */
+	return system(command);
+#endif
+}
+
+
+/*
+ * pg_popen - Custom popen() that lowers file limit in child process.
+ *
+ * This is needed because we may have bumped the soft RLIMIT_NOFILE limit
+ * above its original value. The child process should use the original limit
+ * to avoid issues with programs that use select(2), which can only handle
+ * FDs up to FD_SETSIZE (typically 1024).
+ *
+ * We can't just call UseOriginalOpenFileLimit() before popen() of the stdlib
+ * because popen() opens a file descriptor in the parent process (the parent's
+ * side of the pipe), which would fail if we're already above the original
+ * limit.
+ *
+ * Returns the FILE* for the pipe, or NULL on error.
+ * On success, *child_pid is set to the child's PID (or 0 on Windows where
+ * we use standard popen).
+ */
+static FILE *
+pg_popen(const char *command, const char *mode, pid_t *child_pid)
+{
+#ifndef WIN32
+	int			pipe_fds[2];
+	int			parent_fd;
+	int			child_fd;
+	int			child_target_fd;
+	pid_t		pid;
+	FILE	   *pipe_file;
+	int			save_errno;
+
+	/* Only "r" and "w" modes are supported */
+	if (mode[0] == 'r')
+	{
+		/* Parent reads from child's stdout */
+		parent_fd = 0;
+		child_fd = 1;
+		child_target_fd = STDOUT_FILENO;
+	}
+	else if (mode[0] == 'w')
+	{
+		/* Parent writes to child's stdin */
+		parent_fd = 1;
+		child_fd = 0;
+		child_target_fd = STDIN_FILENO;
+	}
+	else
+	{
+		errno = EINVAL;
+		return NULL;
+	}
+
+	/*
+	 * Set close-on-exec to prevent other concurrently spawned child processes
+	 * from inheriting the pipe file descriptors. This is impossible while
+	 * we're not using threads, but since this is part of adding such
+	 * multithreading support let's pretend that we already have it. For
+	 * systems that don't have pipe2, there's always a risk of race
+	 * conditions, but let's minimize that by quickly setting the
+	 * close-on-exec flag with fnctl.
+	 */
+#ifdef HAVE_PIPE2
+	if (pipe2(pipe_fds, O_CLOEXEC) < 0)
+		return NULL;
+#else
+	if (pipe(pipe_fds) < 0)
+		return NULL;
+	if (fcntl(pipe_fds[0], F_SETFD, FD_CLOEXEC) == -1 ||
+		fcntl(pipe_fds[1], F_SETFD, FD_CLOEXEC) == -1)
+	{
+		save_errno = errno;
+		close(pipe_fds[0]);
+		close(pipe_fds[1]);
+		errno = save_errno;
+		return NULL;
+	}
+#endif
+
+	fflush(NULL);
+
+	pid = fork_for_shell_command();
+	if (pid < 0)
+	{
+		/* fork failed */
+		save_errno = errno;
+		close(pipe_fds[0]);
+		close(pipe_fds[1]);
+		errno = save_errno;
+		return NULL;
+	}
+
+	if (pid == 0)
+	{
+		/* Child process - set up pipe redirection and exec */
+		close(pipe_fds[parent_fd]);
+		if (pipe_fds[child_fd] != child_target_fd)
+		{
+			dup2(pipe_fds[child_fd], child_target_fd);
+			close(pipe_fds[child_fd]);
+		}
+
+		exec_shell_command(command);	/* does not return */
+	}
+
+	/* Parent process */
+	close(pipe_fds[child_fd]);
+	pipe_file = fdopen(pipe_fds[parent_fd], mode);
+	if (pipe_file == NULL)
+	{
+		save_errno = errno;
+		close(pipe_fds[parent_fd]);
+		errno = save_errno;
+		return NULL;
+	}
+
+	*child_pid = pid;
+	return pipe_file;
+#else
+	/* On Windows, just use standard popen - no RLIMIT_NOFILE bumping there */
+	fflush(NULL);
+	*child_pid = 0;
+	return popen(command, mode);
+#endif
+}
+
+/*
+ * pg_pclose - Close a pipe opened by pg_popen().
+ *
+ * On Unix, this closes the FILE* and waits for the child process.
+ * On Windows, this just calls pclose().
+ *
+ * Returns the child's exit status (like pclose), or -1 on error.
+ */
+static int
+pg_pclose(FILE *fp, pid_t child_pid)
+{
+#ifndef WIN32
+	int			status;
+	pid_t		waitresult;
+
+	/* Close the FILE* first */
+	if (fclose(fp) != 0)
+		return -1;
+
+	/* Then wait for the child process, like pclose() does */
+	do
+	{
+		waitresult = waitpid(child_pid, &status, 0);
+	} while (waitresult == (pid_t) -1 && errno == EINTR);
+
+	if (waitresult == child_pid)
+		return status;
+	else
+		return -1;
+#else
+	(void) child_pid;			/* unused on Windows */
+	return pclose(fp);
+#endif
+}
+
 /*
  * Routines that want to initiate a pipe stream should use OpenPipeStream
  * rather than plain popen().  This lets fd.c deal with freeing FDs if
@@ -2745,6 +3141,7 @@ OpenPipeStream(const char *command, const char *mode)
 {
 	FILE	   *file;
 	int			save_errno;
+	pid_t		child_pid;
 
 	DO_DB(elog(LOG, "OpenPipeStream: Allocated %d (%s)",
 			   numAllocatedDescs, command));
@@ -2760,22 +3157,21 @@ OpenPipeStream(const char *command, const char *mode)
 	ReleaseLruFiles();
 
 TryAgain:
-	fflush(NULL);
-	pqsignal(SIGPIPE, SIG_DFL);
+
 	errno = 0;
-	file = popen(command, mode);
+	file = pg_popen(command, mode, &child_pid);
 	save_errno = errno;
 	pqsignal(SIGPIPE, SIG_IGN);
-	errno = save_errno;
 	if (file != NULL)
 	{
 		AllocateDesc *desc = &allocatedDescs[numAllocatedDescs];
 
 		desc->kind = AllocateDescPipe;
-		desc->desc.file = file;
+		desc->desc.pipe.file = file;
+		desc->desc.pipe.pid = child_pid;
 		desc->create_subid = GetCurrentSubTransactionId();
 		numAllocatedDescs++;
-		return desc->desc.file;
+		return desc->desc.pipe.file;
 	}
 
 	if (errno == EMFILE || errno == ENFILE)
@@ -2808,7 +3204,7 @@ FreeDesc(AllocateDesc *desc)
 			result = fclose(desc->desc.file);
 			break;
 		case AllocateDescPipe:
-			result = pclose(desc->desc.file);
+			result = pg_pclose(desc->desc.pipe.file, desc->desc.pipe.pid);
 			break;
 		case AllocateDescDir:
 			result = closedir(desc->desc.dir);
@@ -3060,7 +3456,7 @@ ClosePipeStream(FILE *file)
 	{
 		AllocateDesc *desc = &allocatedDescs[i];
 
-		if (desc->kind == AllocateDescPipe && desc->desc.file == file)
+		if (desc->kind == AllocateDescPipe && desc->desc.pipe.file == file)
 			return FreeDesc(desc);
 	}
 
diff --git a/src/bin/initdb/t/001_initdb.pl b/src/bin/initdb/t/001_initdb.pl
index 1e9543c2585..9a758e26fb7 100644
--- a/src/bin/initdb/t/001_initdb.pl
+++ b/src/bin/initdb/t/001_initdb.pl
@@ -131,7 +131,7 @@ if ($ENV{with_icu} eq 'yes')
 		],
 		'option --icu-locale');
 
-	command_like(
+	command_checks_all(
 		[
 			'initdb', '--no-sync',
 			'--auth' => 'trust',
@@ -145,7 +145,9 @@ if ($ENV{with_icu} eq 'yes')
 			'--lc-time' => 'C',
 			"$tempdir/data4"
 		],
-		qr/^\s+default collation:\s+und\n/ms,
+		0,
+		[qr/^\s+default collation:\s+und\n/ms],
+		[],    # stderr may have LOG messages
 		'options --locale-provider=icu --locale=und --lc-*=C');
 
 	command_fails_like(
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index b0b0cfdaf79..beeee61017d 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -307,6 +307,9 @@
 /* Define to 1 if you have the <pam/pam_appl.h> header file. */
 #undef HAVE_PAM_PAM_APPL_H
 
+/* Define to 1 if you have the `pipe2' function. */
+#undef HAVE_PIPE2
+
 /* Define to 1 if you have the `posix_fadvise' function. */
 #undef HAVE_POSIX_FADVISE
 
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index a1bdefec4a5..4484d557c37 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -186,6 +186,7 @@ extern int	pg_fsync_writethrough(int fd);
 extern int	pg_fdatasync(int fd);
 extern bool pg_file_exists(const char *name);
 extern void pg_flush_data(int fd, pgoff_t offset, pgoff_t nbytes);
+extern int	pg_system(const char *command);
 extern int	pg_truncate(const char *path, pgoff_t length);
 extern void fsync_fname(const char *fname, bool isdir);
 extern int	fsync_fname_ext(const char *fname, bool isdir, bool ignore_perm, int elevel);

base-commit: 6498287696dafc1ebd380ea4eb249124989294d3
-- 
2.52.0

From 94c19d3857102dbf6470d64295fc263b6e68556e Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Sun, 7 Dec 2025 15:06:54 +0100
Subject: [PATCH v9 2/2] Reflect the value of max_safe_fds in
 max_files_per_process

It is currently hard to figure out if max_safe_fds is significantly
lower than max_files_per_process. This starts reflecting the value of
max_safe_fds in max_files_per_process after our limit detection. We
still want to have two separate variables because for the bootstrap or
standalone-backend cases their values differ on purpose.
---
 src/backend/storage/file/fd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 40212d733e9..2d27207ab89 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -1179,6 +1179,7 @@ set_max_safe_fds(void)
 {
 	int			usable_fds;
 	int			already_open;
+	char	   *max_safe_fds_string;
 
 	/*----------
 	 * We want to set max_safe_fds to
@@ -1194,6 +1195,16 @@ set_max_safe_fds(void)
 
 	max_safe_fds = Min(usable_fds, max_files_per_process);
 
+	/*
+	 * Update GUC variable to allow users to see if the result is different
+	 * than what the used value turns out to be different than what they had
+	 * configured.
+	 */
+	max_safe_fds_string = psprintf("%d", max_safe_fds);
+	SetConfigOption("max_files_per_process", max_safe_fds_string,
+					PGC_POSTMASTER, PGC_S_OVERRIDE);
+	pfree(max_safe_fds_string);
+
 	/*
 	 * Take off the FDs reserved for system() etc.
 	 */
-- 
2.52.0