Log connection establishment timings
Hi,
Users wishing to debug slow connection establishment have little
visibility into which steps take the most time. We don't expose any
stats and none of the logging includes durations.
The attached patch logs durations for authentication, forking the
Postgres backend, and the end-to-end connection establishment duration
starting from when the postmaster calls accept() and ending the first
time the forked backend is ready for query.
As an example of how this could be useful, I artificially slowed down
authentication and you can see that that is now visible in logs:
LOG: backend ready for query. pid=86341. socket=10. connection
establishment times (ms): total: 8, fork: 0, authentication: 0
LOG: backend ready for query. pid=86794. socket=10. connection
establishment times (ms): total: 108, fork: 0, authentication: 100
Two notes on implementation:
To make this portable, the timestamps captured in the postmaster
(socket creation time, fork initiation time) are passed through the
ClientSocket and BackendStartupData structures instead of simply saved
in backend local memory inherited by the child process.
Secondly, I used TimestampTz for the times, but it is pretty unclear
to me when instr_time should be used vs TimestampTz for such things.
instr_time says it is more portable, but there are all sorts of places
that use TimestampTz that would probably need to be portable.
Also, INSTR_TIME_SUBTRACT() isn't overflow safe, which must be because
the actual data type of ticks is platform dependent and so the caller
is responsible for not passing it values that would overflow when
subtracted.
- Melanie
Attachments:
v1-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Add-connection-establishment-duration-logging.patchDownload
From f6fa312b605fac31463acc90446aad6afd6933ec Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 3 Dec 2024 18:02:15 -0500
Subject: [PATCH v1] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming conneciton, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
---
src/backend/postmaster/launch_backend.c | 25 +++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 23 +++++++++++++++++++++++
src/backend/utils/adt/timestamp.c | 13 +++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 11 +++++++++++
src/include/libpq/libpq-be.h | 1 +
src/include/miscadmin.h | 3 +++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 3 +++
src/include/utils/timestamp.h | 3 +++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 100 insertions(+)
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 1f2d829ec5a..4c3e3ced13a 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND)
+ ((BackendStartupData *) startup_data)->fork_time = GetCurrentTimestamp();
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time;
+ TimestampTz cur_time = GetCurrentTimestamp();
+
+ conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time,
+ cur_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +632,17 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time;
+ TimestampTz cur_time = GetCurrentTimestamp();
+
+ conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time,
+ cur_time);
+ }
+
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 6f37822c887..642dbb7d880 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1675,7 +1675,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ s.creation_time = GetCurrentTimestamp();
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3383,6 +3390,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ startup_data.fork_time = 0;
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e0a603f42bb..b1fae3cd476 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4280,6 +4280,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4734,6 +4735,28 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections && AmClientBackendProcess())
+ {
+ TimestampTz cur_time = GetCurrentTimestamp();
+ TimestampTz total_duration =
+ TimestampDifferenceMicroseconds(MyClientSocket->creation_time, cur_time);
+
+ ereport(LOG,
+ errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ MyProcPid,
+ (int) MyClientSocket->sock,
+ (long int) total_duration / 1000,
+ (long int) conn_timing.fork_duration / 1000,
+ (long int) conn_timing.auth_duration / 1000));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index 18d7d8a108a..9adaad6ef01 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -1769,6 +1769,19 @@ TimestampDifferenceMilliseconds(TimestampTz start_time, TimestampTz stop_time)
return (long) ((diff + 999) / 1000);
}
+/*
+ * Returns the difference between two timestamps safely as an unsigned int64.
+ */
+uint64
+TimestampDifferenceMicroseconds(TimestampTz start_time, TimestampTz stop_time)
+{
+ /* If start_time is in the future or now, no time has elapsed */
+ if (start_time >= stop_time)
+ return 0;
+
+ return (uint64) stop_time - start_time;
+}
+
/*
* TimestampDifferenceExceeds -- report whether the difference between two
* timestamps is >= a threshold (expressed in milliseconds)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 03a54451ac2..f38a5c34cb1 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 5b657a3f135..2c96ed56849 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,15 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ TimestampTz auth_start_time = 0;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (AmClientBackendProcess())
+ auth_start_time = GetCurrentTimestamp();
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +257,11 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (AmClientBackendProcess())
+ conn_timing.auth_duration = TimestampDifferenceMicroseconds(auth_start_time,
+ GetCurrentTimestamp());
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 9109b2c3344..f0ad28c04ec 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -243,6 +243,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ TimestampTz creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 42a2b38cac9..677afee2350 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -105,6 +105,8 @@ extern PGDLLIMPORT volatile uint32 InterruptHoldoffCount;
extern PGDLLIMPORT volatile uint32 QueryCancelHoldoffCount;
extern PGDLLIMPORT volatile uint32 CritSectionCount;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
/* in tcop/postgres.c */
extern void ProcessInterrupts(void);
@@ -371,6 +373,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmAutoVacuumLauncherProcess() (MyBackendType == B_AUTOVAC_LAUNCHER)
#define AmAutoVacuumWorkerProcess() (MyBackendType == B_AUTOVAC_WORKER)
+#define AmClientBackendProcess() (MyBackendType == B_BACKEND)
#define AmBackgroundWorkerProcess() (MyBackendType == B_BG_WORKER)
#define AmWalSenderProcess() (MyBackendType == B_WAL_SENDER)
#define AmLogicalSlotSyncWorkerProcess() (MyBackendType == B_SLOTSYNC_WORKER)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 24d49a5439e..e7562cb39b8 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -126,6 +126,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ uint64 fork_duration;
+ uint64 auth_duration;
+} ConnectionTiming;
+
/*
* Note: MAX_BACKENDS is limited to 2^18-1 because that's the width reserved
* for buffer references in buf_internals.h. This limitation could be lifted
@@ -155,4 +161,5 @@ typedef enum DispatchOption
extern DispatchOption parse_dispatch_option(const char *name);
+
#endif /* _POSTMASTER_H */
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 993b013afdd..8e248bb8666 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "datatype/timestamp.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,7 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+ TimestampTz fork_time;
} BackendStartupData;
extern void BackendMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed460..77014cd4fff 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -104,6 +104,9 @@ extern void TimestampDifference(TimestampTz start_time, TimestampTz stop_time,
long *secs, int *microsecs);
extern long TimestampDifferenceMilliseconds(TimestampTz start_time,
TimestampTz stop_time);
+
+extern uint64 TimestampDifferenceMicroseconds(TimestampTz start_time,
+ TimestampTz stop_time);
extern bool TimestampDifferenceExceeds(TimestampTz start_time,
TimestampTz stop_time,
int msec);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ce33e55bf1d..a6792ec17a3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -475,6 +475,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
On Mon, 16 Dec 2024 at 22:00, Melanie Plageman
<melanieplageman@gmail.com> wrote:
Users wishing to debug slow connection establishment have little
visibility into which steps take the most time. We don't expose any
stats and none of the logging includes durations.
Two thoughts:
1. Would it make sense to also expose these timings in some pg_stat_xyz view?
2. As a user I'd be curious to know how much of the time is spent on
the network/client vs inside postgres. For example for the scram/sasl
handshake, how much of the authentication_time is spent waiting on the
first "read" after the server has called sendAuthRequest.
Hello,
Le lun. 16 déc. 2024 à 22:00, Melanie Plageman <melanieplageman@gmail.com>
a écrit :
Hi,
Users wishing to debug slow connection establishment have little
visibility into which steps take the most time. We don't expose any
stats and none of the logging includes durations.The attached patch logs durations for authentication, forking the
Postgres backend, and the end-to-end connection establishment duration
starting from when the postmaster calls accept() and ending the first
time the forked backend is ready for query.As an example of how this could be useful, I artificially slowed down
authentication and you can see that that is now visible in logs:LOG: backend ready for query. pid=86341. socket=10. connection
establishment times (ms): total: 8, fork: 0, authentication: 0
LOG: backend ready for query. pid=86794. socket=10. connection
establishment times (ms): total: 108, fork: 0, authentication: 100Two notes on implementation:
To make this portable, the timestamps captured in the postmaster
(socket creation time, fork initiation time) are passed through the
ClientSocket and BackendStartupData structures instead of simply saved
in backend local memory inherited by the child process.Secondly, I used TimestampTz for the times, but it is pretty unclear
to me when instr_time should be used vs TimestampTz for such things.
instr_time says it is more portable, but there are all sorts of places
that use TimestampTz that would probably need to be portable.
Also, INSTR_TIME_SUBTRACT() isn't overflow safe, which must be because
the actual data type of ticks is platform dependent and so the caller
is responsible for not passing it values that would overflow when
subtracted.
Just a quick note. I like this patch a lot and it would help to diagnose
some situations our customers may have. I tried the patch on my laptop, and
it works fine. I'll find some time to read the code as well, but in the
meantime, this looks like a nice thing to have in PostgreSQL.
Thanks Melanie.
--
Guillaume.
Hi,
On Mon, Jan 06, 2025 at 10:08:36PM +0100, Guillaume Lelarge wrote:
Hello,
Le lun. 16 d�c. 2024 � 22:00, Melanie Plageman <melanieplageman@gmail.com>
a �crit :Hi,
Users wishing to debug slow connection establishment have little
visibility into which steps take the most time. We don't expose any
stats and none of the logging includes durations.The attached patch logs durations for authentication, forking the
Postgres backend, and the end-to-end connection establishment duration
starting from when the postmaster calls accept() and ending the first
time the forked backend is ready for query.As an example of how this could be useful, I artificially slowed down
authentication and you can see that that is now visible in logs:LOG: backend ready for query. pid=86341. socket=10. connection
establishment times (ms): total: 8, fork: 0, authentication: 0
LOG: backend ready for query. pid=86794. socket=10. connection
establishment times (ms): total: 108, fork: 0, authentication: 100Two notes on implementation:
To make this portable, the timestamps captured in the postmaster
(socket creation time, fork initiation time) are passed through the
ClientSocket and BackendStartupData structures instead of simply saved
in backend local memory inherited by the child process.Secondly, I used TimestampTz for the times, but it is pretty unclear
to me when instr_time should be used vs TimestampTz for such things.
instr_time says it is more portable, but there are all sorts of places
that use TimestampTz that would probably need to be portable.
Also, INSTR_TIME_SUBTRACT() isn't overflow safe, which must be because
the actual data type of ticks is platform dependent and so the caller
is responsible for not passing it values that would overflow when
subtracted.Just a quick note. I like this patch a lot and it would help to diagnose
some situations our customers may have. I tried the patch on my laptop, and
it works fine. I'll find some time to read the code as well, but in the
meantime, this looks like a nice thing to have in PostgreSQL.
Also +1 on the idea.
The patch needed a rebase due to 34486b6092e. I did it in v2 attached (it's
a minor rebase due to the AmRegularBackendProcess() introduction in miscadmin.h).
v2 could rely on AmRegularBackendProcess() instead of AmClientBackendProcess()
but I kept it with AmClientBackendProcess() to reduce "my" changes as compared to
v1.
Regarding the TimestampTz vs instr_time choice, we have things like:
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time;
+ TimestampTz cur_time = GetCurrentTimestamp();
+
+ conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time,
+ cur_time);
but looking at TimestampDifferenceMicroseconds():
"
/* If start_time is in the future or now, no time has elapsed */
if (start_time >= stop_time)
return 0;
"
I think that it can happen due to time changes.
So with TimestampTz, we would:
1. return 0 if we moved the time backward
2. provide an inflated duration including the time jump (if the time move
forward).
But with instr_time (and on systems that support CLOCK_MONOTONIC) then
pg_clock_gettime_ns() should not be affected by system time change IIUC.
Though time changes are "rare", given the fact that those metrics could provide
"inaccurate" measurements during that particular moment (time change) then it
might be worth considering instr_time instead for this particular metric.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v2-0001-Add-connection-establishment-duration-logging.patchtext/x-diff; charset=us-asciiDownload
From 570f72f4bc1eb71216f898246d4c752034422c24 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 3 Dec 2024 18:02:15 -0500
Subject: [PATCH v2] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming conneciton, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
---
src/backend/postmaster/launch_backend.c | 25 +++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 23 +++++++++++++++++++++++
src/backend/utils/adt/timestamp.c | 13 +++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 11 +++++++++++
src/include/libpq/libpq-be.h | 1 +
src/include/miscadmin.h | 3 +++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 3 +++
src/include/utils/timestamp.h | 3 +++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 100 insertions(+)
34.0% src/backend/postmaster/
28.4% src/backend/tcop/
10.5% src/backend/utils/adt/
13.0% src/backend/utils/init/
3.3% src/include/postmaster/
3.4% src/include/utils/
6.5% src/include/
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index a97a1eda6da..009c4083906 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND)
+ ((BackendStartupData *) startup_data)->fork_time = GetCurrentTimestamp();
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time;
+ TimestampTz cur_time = GetCurrentTimestamp();
+
+ conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time,
+ cur_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +632,17 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time;
+ TimestampTz cur_time = GetCurrentTimestamp();
+
+ conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time,
+ cur_time);
+ }
+
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5f615d0f605..b60f5d4c7d8 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1676,7 +1676,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ s.creation_time = GetCurrentTimestamp();
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3458,6 +3465,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ startup_data.fork_time = 0;
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 5655348a2e2..cbd952f2405 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4151,6 +4151,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4605,6 +4606,28 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections && AmClientBackendProcess())
+ {
+ TimestampTz cur_time = GetCurrentTimestamp();
+ TimestampTz total_duration =
+ TimestampDifferenceMicroseconds(MyClientSocket->creation_time, cur_time);
+
+ ereport(LOG,
+ errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ MyProcPid,
+ (int) MyClientSocket->sock,
+ (long int) total_duration / 1000,
+ (long int) conn_timing.fork_duration / 1000,
+ (long int) conn_timing.auth_duration / 1000));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index ba9bae05069..67827613bfc 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -1769,6 +1769,19 @@ TimestampDifferenceMilliseconds(TimestampTz start_time, TimestampTz stop_time)
return (long) ((diff + 999) / 1000);
}
+/*
+ * Returns the difference between two timestamps safely as an unsigned int64.
+ */
+uint64
+TimestampDifferenceMicroseconds(TimestampTz start_time, TimestampTz stop_time)
+{
+ /* If start_time is in the future or now, no time has elapsed */
+ if (start_time >= stop_time)
+ return 0;
+
+ return (uint64) stop_time - start_time;
+}
+
/*
* TimestampDifferenceExceeds -- report whether the difference between two
* timestamps is >= a threshold (expressed in milliseconds)
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 01bb6a410cb..38aba9d3435 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -189,9 +189,15 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ TimestampTz auth_start_time = 0;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (AmClientBackendProcess())
+ auth_start_time = GetCurrentTimestamp();
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -250,6 +256,11 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (AmClientBackendProcess())
+ conn_timing.auth_duration = TimestampDifferenceMicroseconds(auth_start_time,
+ GetCurrentTimestamp());
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 2f6c29200ba..3898735267c 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -252,6 +252,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ TimestampTz creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index d016a9c9248..76dd27de26d 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -105,6 +105,8 @@ extern PGDLLIMPORT volatile uint32 InterruptHoldoffCount;
extern PGDLLIMPORT volatile uint32 QueryCancelHoldoffCount;
extern PGDLLIMPORT volatile uint32 CritSectionCount;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
/* in tcop/postgres.c */
extern void ProcessInterrupts(void);
@@ -379,6 +381,7 @@ extern PGDLLIMPORT BackendType MyBackendType;
#define AmRegularBackendProcess() (MyBackendType == B_BACKEND)
#define AmAutoVacuumLauncherProcess() (MyBackendType == B_AUTOVAC_LAUNCHER)
#define AmAutoVacuumWorkerProcess() (MyBackendType == B_AUTOVAC_WORKER)
+#define AmClientBackendProcess() (MyBackendType == B_BACKEND)
#define AmBackgroundWorkerProcess() (MyBackendType == B_BG_WORKER)
#define AmWalSenderProcess() (MyBackendType == B_WAL_SENDER)
#define AmLogicalSlotSyncWorkerProcess() (MyBackendType == B_SLOTSYNC_WORKER)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index 188a06e2379..6826e1532c3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -126,6 +126,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ uint64 fork_duration;
+ uint64 auth_duration;
+} ConnectionTiming;
+
/*
* Note: MAX_BACKENDS is limited to 2^18-1 because that's the width reserved
* for buffer references in buf_internals.h. This limitation could be lifted
@@ -155,4 +161,5 @@ typedef enum DispatchOption
extern DispatchOption parse_dispatch_option(const char *name);
+
#endif /* _POSTMASTER_H */
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 01baf4aad75..b08521c2043 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "datatype/timestamp.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,7 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+ TimestampTz fork_time;
} BackendStartupData;
extern void BackendMain(char *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index d26f023fb87..90b64b45dc5 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -104,6 +104,9 @@ extern void TimestampDifference(TimestampTz start_time, TimestampTz stop_time,
long *secs, int *microsecs);
extern long TimestampDifferenceMilliseconds(TimestampTz start_time,
TimestampTz stop_time);
+
+extern uint64 TimestampDifferenceMicroseconds(TimestampTz start_time,
+ TimestampTz stop_time);
extern bool TimestampDifferenceExceeds(TimestampTz start_time,
TimestampTz stop_time,
int msec);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d5aa5c295ae..29fc1bba01a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -480,6 +480,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
On Mon, Jan 20, 2025 at 7:01 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
Though time changes are "rare", given the fact that those metrics could provide
"inaccurate" measurements during that particular moment (time change) then it
might be worth considering instr_time instead for this particular metric.
+1, I think a CLOCK_MONOTONIC source should be used for this if we've got it.
--
For the EXEC_BACKEND case (which, to be honest, I don't know much
about), I originally wondered if the fork_duration should include any
of the shared memory manipulations or library reloads that are done to
match Unix behavior. But I think I prefer the way the patch does it.
Can the current timestamp be recorded right at the beginning of
SubPostmasterMain(), to avoid counting the time setting up GUCs and
reading the variables file, or do we have to wait?
nit: conn_timing is currently declared in the "interrupts and crit
section" part of miscadmin.h; should it be moved down to the
general-purpose globals?
Thanks,
--Jacob
On Mon, Dec 16, 2024 at 6:26 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
Two thoughts:
1. Would it make sense to also expose these timings in some pg_stat_xyz view?
So, right now there isn't an obvious place this would fit. There are
dynamic statistics views that are per connection -- like
pg_stat_ssl/gssapi. We could perhaps add authentication duration to
one of these. But I don't think this really helps users with the use
case I imagine for connection establishment logging. I thought that
people would look at the logs to see how their connection times have
changed over time -- for example if authentication started taking
longer. This also means a cumulative statistics view wouldn't be very
helpful. We could add a view to calculate connection establishment
median, standard deviation, average, etc, but that doesn't seem as
useful as the history of timings and how they have changed over that
time.
2. As a user I'd be curious to know how much of the time is spent on
the network/client vs inside postgres. For example for the scram/sasl
handshake, how much of the authentication_time is spent waiting on the
first "read" after the server has called sendAuthRequest.
Well, with the current patch you can compare authentication duration
to total connection establishment duration to see how much time was
spent on authentication vs the rest of connection establishment.
Or do you mean adding another more granular level of timing: after
sendAuthRequest until PerformAuthentication() returns?
I'm not totally clear what "read" means in this context.
- Melanie
Thanks for taking a look!
On Mon, Jan 20, 2025 at 10:01 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
The patch needed a rebase due to 34486b6092e. I did it in v2 attached (it's
a minor rebase due to the AmRegularBackendProcess() introduction in miscadmin.h).v2 could rely on AmRegularBackendProcess() instead of AmClientBackendProcess()
but I kept it with AmClientBackendProcess() to reduce "my" changes as compared to
v1.
Thanks for doing this! I have implemented your suggestion in attached v3.
Regarding the TimestampTz vs instr_time choice, we have things like:
< -- snip -- >
So with TimestampTz, we would:
1. return 0 if we moved the time backward
2. provide an inflated duration including the time jump (if the time move
forward).But with instr_time (and on systems that support CLOCK_MONOTONIC) then
pg_clock_gettime_ns() should not be affected by system time change IIUC.Though time changes are "rare", given the fact that those metrics could provide
"inaccurate" measurements during that particular moment (time change) then it
might be worth considering instr_time instead for this particular metric.
Great point. This all makes sense. I've switched to using instr_time in v3.
- Melanie
Attachments:
v3-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v3-0001-Add-connection-establishment-duration-logging.patchDownload
From 660e2f3846fd1af74956a45559da5ea17d5dbc92 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 25 Feb 2025 13:08:48 -0500
Subject: [PATCH v3] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming conneciton, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
src/backend/postmaster/launch_backend.c | 20 ++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 24 ++++++++++++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 13 +++++++++++++
src/include/libpq/libpq-be.h | 2 ++
src/include/miscadmin.h | 2 ++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 80 insertions(+)
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..37b31069120 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND)
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,14 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..880f491a9f7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1685,7 +1685,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ INSTR_TIME_SET_CURRENT(s.creation_time);
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3511,6 +3518,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ INSTR_TIME_SET_ZERO(startup_data.fork_time);
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..abd7e648657 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,29 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections && AmRegularBackendProcess())
+ {
+ instr_time total_duration;
+
+ INSTR_TIME_SET_CURRENT(total_duration);
+ INSTR_TIME_SUBTRACT(total_duration, MyClientSocket->creation_time);
+
+ ereport(LOG,
+ errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ MyProcPid,
+ (int) MyClientSocket->sock,
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..c262849b9ac 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,15 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (AmRegularBackendProcess())
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +257,13 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (AmRegularBackendProcess())
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.auth_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.auth_duration, auth_start_time);
+ }
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..b16ad69efff 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
@@ -252,6 +253,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ instr_time creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..71a3f5f644d 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -126,6 +127,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..d5d13bf140a 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -37,6 +37,7 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+ instr_time fork_time;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1649f5e1011..34d2daa0c9b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
Thanks for taking a look!
On Mon, Jan 20, 2025 at 12:53 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
On Mon, Jan 20, 2025 at 7:01 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:Though time changes are "rare", given the fact that those metrics could provide
"inaccurate" measurements during that particular moment (time change) then it
might be worth considering instr_time instead for this particular metric.+1, I think a CLOCK_MONOTONIC source should be used for this if we've got it.
Done in v3 (see [1]/messages/by-id/CAAKRu_YrNsA7-v5L9d318XZu+tPqcxp+ctCGy2EGYrSt69ON=A@mail.gmail.com).
For the EXEC_BACKEND case (which, to be honest, I don't know much
about), I originally wondered if the fork_duration should include any
of the shared memory manipulations or library reloads that are done to
match Unix behavior. But I think I prefer the way the patch does it.
You mean if the EXEC_BACKEND not defined version should calculate the
end of fork_duration basically at the end of
postmaster_child_launch()?
Can the current timestamp be recorded right at the beginning of
SubPostmasterMain(), to avoid counting the time setting up GUCs and
reading the variables file, or do we have to wait?
We actually don't have the startup data until after we
read_backend_variables(), so I did it as soon as I could after that.
You are right that this will include timing from some extra steps.
But, some of these steps are overhead unique to the slow process of
"forking" a backend when EXEC_BACKEND is defined anyway, so it
probably makes sense for them to be included in the timing of "fork"
for these backends.
nit: conn_timing is currently declared in the "interrupts and crit
section" part of miscadmin.h; should it be moved down to the
general-purpose globals?
Whoops, it would help if I read comments and stuff. Thanks! Fixed in v3 in [1]/messages/by-id/CAAKRu_YrNsA7-v5L9d318XZu+tPqcxp+ctCGy2EGYrSt69ON=A@mail.gmail.com.
- Melanie
[1]: /messages/by-id/CAAKRu_YrNsA7-v5L9d318XZu+tPqcxp+ctCGy2EGYrSt69ON=A@mail.gmail.com
On Tue, Feb 25, 2025 at 3:23 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
Thanks for doing this! I have implemented your suggestion in attached v3.
I missed an include in the EXEC_BACKEND not defined case. attached v4
is fixed up.
- Melanie
Attachments:
v4-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v4-0001-Add-connection-establishment-duration-logging.patchDownload
From c493f0be7243bfe965d3493f07982ea1072d89b7 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 25 Feb 2025 13:08:48 -0500
Subject: [PATCH v4] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming conneciton, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
src/backend/postmaster/launch_backend.c | 20 ++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 24 ++++++++++++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 13 +++++++++++++
src/include/libpq/libpq-be.h | 2 ++
src/include/miscadmin.h | 2 ++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 3 +++
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 82 insertions(+)
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..37b31069120 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND)
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,14 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..880f491a9f7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1685,7 +1685,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ INSTR_TIME_SET_CURRENT(s.creation_time);
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3511,6 +3518,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ INSTR_TIME_SET_ZERO(startup_data.fork_time);
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..abd7e648657 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,29 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections && AmRegularBackendProcess())
+ {
+ instr_time total_duration;
+
+ INSTR_TIME_SET_CURRENT(total_duration);
+ INSTR_TIME_SUBTRACT(total_duration, MyClientSocket->creation_time);
+
+ ereport(LOG,
+ errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ MyProcPid,
+ (int) MyClientSocket->sock,
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..c262849b9ac 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,15 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (AmRegularBackendProcess())
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +257,13 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (AmRegularBackendProcess())
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.auth_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.auth_duration, auth_start_time);
+ }
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..b16ad69efff 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
@@ -252,6 +253,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ instr_time creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..71a3f5f644d 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -126,6 +127,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..deb59c78425 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,7 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+ instr_time fork_time;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1649f5e1011..34d2daa0c9b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
On 2025/02/26 6:36, Melanie Plageman wrote:
On Tue, Feb 25, 2025 at 3:23 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:Thanks for doing this! I have implemented your suggestion in attached v3.
I missed an include in the EXEC_BACKEND not defined case. attached v4
is fixed up.
Thanks for updating the patch!
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND)
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
When log_connections is enabled, walsender connections are also logged.
However, with the patch, it seems the connection time for walsenders isn't captured.
Is this intentional?
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?
+ ereport(LOG,
+ errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ MyProcPid,
+ (int) MyClientSocket->sock,
Why expose the socket's file descriptor? I'm not sure how users would use that information.
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Hi,
On Wed, Feb 26, 2025 at 01:46:19PM +0900, Fujii Masao wrote:
On 2025/02/26 6:36, Melanie Plageman wrote:
On Tue, Feb 25, 2025 at 3:23 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:Thanks for doing this! I have implemented your suggestion in attached v3.
Thanks for the new patch version!
+ /* Capture time Postmaster initiates fork for logging */ + if (child_type == B_BACKEND) + INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);When log_connections is enabled, walsender connections are also logged.
However, with the patch, it seems the connection time for walsenders isn't captured.
Is this intentional?
Good point. I'm tempted to say that it should also be, specially because a
connection done as "psql replication=database" is of "walsender" backend type and
would not be reported.
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?
+1, I also think it's a good idea to let users decide if they want the timing
measurement overhead (and it's common practice with track_io_timing,
track_wal_io_timing, the newly track_cost_delay_timing for example)
+ ereport(LOG, + errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld", + MyProcPid, + (int) MyClientSocket->sock,Why expose the socket's file descriptor? I'm not sure how users would use that information.
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Yeah, we would get things like:
[1111539]: LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0
[1111539]: LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0
[1111539]: LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0
[1111539]: LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0
I also wonder if "backend ready for query" is worth it. Maybe something like:
2025-02-26 06:44:23.265 UTC [1111539]LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0 LOG: connection establishment times (ms): total: 2, fork: 0, authentication: 0
would be good enough?
A few random comments:
=== 1
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
As it's all about instr_time, I wonder if we could use an enum + array instead.
That's probably just a matter of taste but that sounds more flexible to extend
(should we want to add more timing in the future).
=== 2
+ConnectionTiming conn_timing = {0};
There is no padding in ConnectionTiming and anyway we just access its fields
so that's ok to initialize that way.
=== 3
Add a few words in the log_connections GUC doc? (anyway we will have to if
Fujii-san idea above about the timing is implemented)
=== 4
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (child_type == B_BACKEND)
+ {
+ INSTR_TIME_SET_CURRENT(conn_timing.fork_duration);
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration,
+ ((BackendStartupData *) startup_data)->fork_time);
+ }
worth to add a helper function to avoid code duplication?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On 26/02/2025 08:41, Bertrand Drouvot wrote:
Hi,
On Wed, Feb 26, 2025 at 01:46:19PM +0900, Fujii Masao wrote:
On 2025/02/26 6:36, Melanie Plageman wrote:
On Tue, Feb 25, 2025 at 3:23 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:Thanks for doing this! I have implemented your suggestion in attached v3.
Thanks for the new patch version!
+1
+ /* Capture time Postmaster initiates fork for logging */ + if (child_type == B_BACKEND) + INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);When log_connections is enabled, walsender connections are also logged.
However, with the patch, it seems the connection time for walsenders isn't captured.
Is this intentional?Good point. I'm tempted to say that it should also be, specially because a
connection done as "psql replication=database" is of "walsender" backend type and
would not be reported.
Agreed.
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?+1, I also think it's a good idea to let users decide if they want the timing
measurement overhead (and it's common practice with track_io_timing,
track_wal_io_timing, the newly track_cost_delay_timing for example)
track_connection_delay_timing ? I'm fine with this, but I'm a bit afraid
that it will lead us to an awful lot of GUCs for simple things.
+ ereport(LOG, + errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld", + MyProcPid, + (int) MyClientSocket->sock,Why expose the socket's file descriptor? I'm not sure how users would use that information.
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Yeah, we would get things like:
[1111539] LOG: connection received: host=[local]
[1111539] LOG: connection authenticated: user="postgres" method=trust (/home/postgres/postgresql/pg_installed/pg18/data/pg_hba.conf:117)
[1111539] LOG: connection authorized: user=postgres database=postgres application_name=psql
[1111539] LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0I also wonder if "backend ready for query" is worth it. Maybe something like:
2025-02-26 06:44:23.265 UTC [1111539] LOG: connection establishment times (ms): total: 2, fork: 0, authentication: 0
would be good enough?
Sounds definitely better to me.
A few random comments:
=== 1
+typedef struct ConnectionTiming +{ + instr_time fork_duration; + instr_time auth_duration; +} ConnectionTiming;As it's all about instr_time, I wonder if we could use an enum + array instead.
That's probably just a matter of taste but that sounds more flexible to extend
(should we want to add more timing in the future).=== 2
+ConnectionTiming conn_timing = {0};
There is no padding in ConnectionTiming and anyway we just access its fields
so that's ok to initialize that way.=== 3
Add a few words in the log_connections GUC doc? (anyway we will have to if
Fujii-san idea above about the timing is implemented)=== 4
+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + } + /* Close the postmaster's sockets */ ClosePostmasterPorts(child_type == B_LOGGER);@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + }worth to add a helper function to avoid code duplication?
Regards,
--
Guillaume Lelarge
Consultant
https://dalibo.com
Thanks for the review!
On Tue, Feb 25, 2025 at 11:46 PM Fujii Masao
<masao.fujii@oss.nttdata.com> wrote:
On 2025/02/26 6:36, Melanie Plageman wrote:
On Tue, Feb 25, 2025 at 3:23 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:+ /* Capture time Postmaster initiates fork for logging */ + if (child_type == B_BACKEND) + INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);When log_connections is enabled, walsender connections are also logged.
However, with the patch, it seems the connection time for walsenders isn't captured.
Is this intentional?
Ah, great point. It was not intentional. I've added a check for wal
sender to the attached v5.
Are these the only backend types that establish connections from outside?
This makes me wonder if I don't need these checks (for backend type)
before capturing the current time in PerformAuthentication() -- that
is, if they are performing authentication, aren't they inherently one
of these backend types?
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?
Ah yes, I did this intentionally originally because I thought someone
might change the value of log_connections in between the start and end
of the duration. I now see you cannot change log_connections after
connection start. So, I can guard these behind log_connections (done
in attached v5).
I hesitate to have a separate guc controlling whether or not we log
connection timing. If we add far more instances of getting the current
time, then perhaps it makes sense. But, as it is, we are adding six
system calls that take on the order of nanoseconds (esp if using
clock_gettime()), whereas emitting each log message -- which
log_connections allows -- will take on the order of micro or even
milliseconds.
+ ereport(LOG, + errmsg("backend ready for query. pid=%d. socket=%d. connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld", + MyProcPid, + (int) MyClientSocket->sock,Why expose the socket's file descriptor? I'm not sure how users would use that information.
I originally included the socket fd because I thought we might end up
printing the times instead of the durations and then users would have
to parse them to get the durations and would need a way to uniquely
identify a connection. This would ideally be a combination of client
address, client port, server address, server port -- but those are
harder to print out (due to IP versions, etc) and harder to parse.
Also, I did notice other places printing the socket (like
BackendStartup() after forking).
Since this version is just printing one message, I have removed the socket fd.
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Ah, great point. I've removed that from the log message in the attached version
- Melanie
Attachments:
v5-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v5-0001-Add-connection-establishment-duration-logging.patchDownload
From 743d488161758ff923573cc65f3ac437feafd738 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 25 Feb 2025 13:08:48 -0500
Subject: [PATCH v5] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
src/backend/postmaster/launch_backend.c | 23 +++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 12 ++++++++++++
src/include/libpq/libpq-be.h | 2 ++
src/include/miscadmin.h | 2 ++
src/include/portability/instr_time.h | 9 +++++++++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 5 +++++
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 92 insertions(+)
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..c482cb299f7 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,11 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +245,15 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +632,15 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..880f491a9f7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1685,7 +1685,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ INSTR_TIME_SET_CURRENT(s.creation_time);
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3511,6 +3518,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ INSTR_TIME_SET_ZERO(startup_data.fork_time);
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..61d4ae2474d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(MyClientSocket->creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..30eed2e85f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,16 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +258,11 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..b16ad69efff 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
@@ -252,6 +253,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ instr_time creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..71a3f5f644d 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -126,6 +127,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..7d9c43ce77b 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,9 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /* Time at which the postmaster initiates a fork of a backend process */
+ instr_time fork_time;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfbab589d61..566b0d8d73b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
Thanks for the continued review!
On Wed, Feb 26, 2025 at 2:41 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Wed, Feb 26, 2025 at 01:46:19PM +0900, Fujii Masao wrote:
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?+1, I also think it's a good idea to let users decide if they want the timing
measurement overhead (and it's common practice with track_io_timing,
track_wal_io_timing, the newly track_cost_delay_timing for example)
It seems to me like the extra timing collected and the one additional
log message isn't enough overhead to justify its own guc (for now).
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Yeah, we would get things like:
[1111539] LOG: connection received: host=[local]
[1111539] LOG: connection authenticated: user="postgres" method=trust (/home/postgres/postgresql/pg_installed/pg18/data/pg_hba.conf:117)
[1111539] LOG: connection authorized: user=postgres database=postgres application_name=psql
[1111539] LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0I also wonder if "backend ready for query" is worth it. Maybe something like:
2025-02-26 06:44:23.265 UTC [1111539] LOG: connection establishment times (ms): total: 2, fork: 0, authentication: 0
would be good enough?
Yes, thank you. v5 attached in [1]/messages/by-id/CAAKRu_Y9sgZAWCiQoHtpwx6Mv28fBGav5ztrWyeSrx+B=ACN6g@mail.gmail.com changes the wording as you recommend..
+typedef struct ConnectionTiming +{ + instr_time fork_duration; + instr_time auth_duration; +} ConnectionTiming;As it's all about instr_time, I wonder if we could use an enum + array instead.
That's probably just a matter of taste but that sounds more flexible to extend
(should we want to add more timing in the future).
I think we can change it later if we add many more. For now I prefer
the clarity of accessing members by name. Especially because we don't
have any code yet that loops through all of them or anything like
that.
+ConnectionTiming conn_timing = {0};
There is no padding in ConnectionTiming and anyway we just access its fields
so that's ok to initialize that way.
Yes, this properly zero initializes the struct. In fact it shouldn't
be needed since a global like this should be zero initialized. But all
of the globals defined above conn_timing zero initialize themselves,
so I thought I would be consistent with them.
Add a few words in the log_connections GUC doc? (anyway we will have to if
Fujii-san idea above about the timing is implemented)
I forgot to do this in v5 attached in [1]/messages/by-id/CAAKRu_Y9sgZAWCiQoHtpwx6Mv28fBGav5ztrWyeSrx+B=ACN6g@mail.gmail.com. Let me go ahead and do this next.
+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + } + /* Close the postmaster's sockets */ ClosePostmasterPorts(child_type == B_LOGGER);@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + }worth to add a helper function to avoid code duplication?
I've added INSTR_TIME_GET_DURATION_SINCE(start_time). Which I like
because it seems generally useful. It does not however cut down on
LOC, so I'm somewhat on the fence.
- Melanie
[1]: /messages/by-id/CAAKRu_Y9sgZAWCiQoHtpwx6Mv28fBGav5ztrWyeSrx+B=ACN6g@mail.gmail.com
On Wed, Feb 26, 2025 at 1:45 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
Thanks for the continued review!
On Wed, Feb 26, 2025 at 2:41 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:Add a few words in the log_connections GUC doc? (anyway we will have to if
Fujii-san idea above about the timing is implemented)I forgot to do this in v5 attached in [1]. Let me go ahead and do this next.
I took a stab at this in attached v6. I feel that what I have is a bit
stilted, but I'm not sure how to fix it.
- Melanie
Attachments:
v6-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v6-0001-Add-connection-establishment-duration-logging.patchDownload
From 990f6841c3a6795d315821278d985cf0c5632f5e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 25 Feb 2025 13:08:48 -0500
Subject: [PATCH v6] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
ci-os-only:
---
doc/src/sgml/config.sgml | 1 +
src/backend/postmaster/launch_backend.c | 23 +++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 ++++++++
src/backend/tcop/postgres.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 12 ++++++++++++
src/include/libpq/libpq-be.h | 2 ++
src/include/miscadmin.h | 2 ++
src/include/portability/instr_time.h | 9 +++++++++
src/include/postmaster/postmaster.h | 7 +++++++
src/include/tcop/backend_startup.h | 5 +++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 93 insertions(+)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e55700f35b8..4861796654b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7325,6 +7325,7 @@ local0.* /var/log/postgresql
Causes each attempted connection to the server to be logged,
as well as successful completion of both client authentication (if
necessary) and authorization.
+ Successful connections log the connection establishment duration.
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..c482cb299f7 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,11 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +245,15 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +632,15 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if (Log_connections &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..880f491a9f7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1685,7 +1685,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ INSTR_TIME_SET_CURRENT(s.creation_time);
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3511,6 +3518,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ INSTR_TIME_SET_ZERO(startup_data.fork_time);
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..61d4ae2474d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(MyClientSocket->creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..30eed2e85f7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,16 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +258,11 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (Log_connections &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (Log_connections)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..b16ad69efff 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
@@ -252,6 +253,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ instr_time creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..71a3f5f644d 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -126,6 +127,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..7d9c43ce77b 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,9 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /* Time at which the postmaster initiates a fork of a backend process */
+ instr_time fork_time;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfbab589d61..566b0d8d73b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
Hi,
On Wed, Feb 26, 2025 at 01:45:39PM -0500, Melanie Plageman wrote:
Thanks for the continued review!
On Wed, Feb 26, 2025 at 2:41 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Wed, Feb 26, 2025 at 01:46:19PM +0900, Fujii Masao wrote:
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?+1, I also think it's a good idea to let users decide if they want the timing
measurement overhead (and it's common practice with track_io_timing,
track_wal_io_timing, the newly track_cost_delay_timing for example)It seems to me like the extra timing collected and the one additional
log message isn't enough overhead to justify its own guc (for now).
Agree. IIUC, I think that Fujii-san's idea was to extend log_connections with
a new option "timing" (i.e move it from ConfigureNamesBool to say
ConfigureNamesEnum with say on, off and timing?). I think that's a good idea.
I just did a quick check and changing a GUC from ConfigureNamesBool to
ConfigureNamesEnum is something that has already been done in the past (see
240067b3b0f and ffd37740ee6 for example).
In my previous up-thead message, I did not mean to suggest to add a new GUC,
just saying that when new "timing" is measured then users have the choice to
enable or disable it.
Including the PID seems unnecessary since it's already available via log_line_prefix with %p?
Yeah, we would get things like:
[1111539] LOG: connection received: host=[local]
[1111539] LOG: connection authenticated: user="postgres" method=trust (/home/postgres/postgresql/pg_installed/pg18/data/pg_hba.conf:117)
[1111539] LOG: connection authorized: user=postgres database=postgres application_name=psql
[1111539] LOG: backend ready for query. pid=1111539. socket=9. connection establishment times (ms): total: 2, fork: 0, authentication: 0I also wonder if "backend ready for query" is worth it. Maybe something like:
2025-02-26 06:44:23.265 UTC [1111539] LOG: connection establishment times (ms): total: 2, fork: 0, authentication: 0
would be good enough?
Yes, thank you. v5 attached in [1] changes the wording as you recommend..
Thanks for the updated version!
+typedef struct ConnectionTiming +{ + instr_time fork_duration; + instr_time auth_duration; +} ConnectionTiming;As it's all about instr_time, I wonder if we could use an enum + array instead.
That's probably just a matter of taste but that sounds more flexible to extend
(should we want to add more timing in the future).I think we can change it later if we add many more. For now I prefer
the clarity of accessing members by name. Especially because we don't
have any code yet that loops through all of them or anything like
that.
Yeah makes sense.
Add a few words in the log_connections GUC doc? (anyway we will have to if
Fujii-san idea above about the timing is implemented)I took a stab at this in attached v6. I feel that what I have is a bit
stilted, but I'm not sure how to fix it.
Yeah, that might be easier to reason about if we're going with Fujii-san
suggestion to extend log_connections with a new option?
+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + } + /* Close the postmaster's sockets */ ClosePostmasterPorts(child_type == B_LOGGER);@@ -618,6 +630,14 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);+ /* Calculate total fork duration in child backend for logging */ + if (child_type == B_BACKEND) + { + INSTR_TIME_SET_CURRENT(conn_timing.fork_duration); + INSTR_TIME_SUBTRACT(conn_timing.fork_duration, + ((BackendStartupData *) startup_data)->fork_time); + }worth to add a helper function to avoid code duplication?
I've added INSTR_TIME_GET_DURATION_SINCE(start_time). Which I like
because it seems generally useful.
Great idea! Could probably be used in other places but I did not check and
it's outside the scope of this thread anyway.
It does not however cut down on LOC, so I'm somewhat on the fence.
I think that's somehow also around code maintenance (not only LOC), say for example
if we want to add more "child_type" in the check (no need to remember to update both
locations).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Thu, Feb 27, 2025 at 1:50 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
Agree. IIUC, I think that Fujii-san's idea was to extend log_connections with
a new option "timing" (i.e move it from ConfigureNamesBool to say
ConfigureNamesEnum with say on, off and timing?). I think that's a good idea.I just did a quick check and changing a GUC from ConfigureNamesBool to
ConfigureNamesEnum is something that has already been done in the past (see
240067b3b0f and ffd37740ee6 for example).In my previous up-thead message, I did not mean to suggest to add a new GUC,
just saying that when new "timing" is measured then users have the choice to
enable or disable it.
I was just talking to Andres off-list and he mentioned that the volume
of log_connections messages added in recent releases can really be a
problem for users. He said ideally we would emit one message which
consolidated these (and make sure we did so for failed connections too
detailing the successfully completed stages).
However, since that is a bigger project (with more refactoring, etc),
he suggested that we change log_connections to a GUC_LIST
(ConfigureNamesString) with options like "none", "received,
"authenticated", "authorized", "all".
Then we could add one like "established" for the final message and
timings my patch set adds. I think the overhead of an additional log
message being emitted probably outweighs the overhead of taking those
additional timings.
String GUCs are a lot more work than enum GUCs, so I was thinking if
there is a way to do it as an enum.
I think we want the user to be able to specify a list of all the log
messages they want included, not just have each one include the
previous ones. So, then it probably has to be a list right? There is
no good design that would fit as an enum.
I've added INSTR_TIME_GET_DURATION_SINCE(start_time). Which I like
because it seems generally useful.Great idea! Could probably be used in other places but I did not check and
it's outside the scope of this thread anyway.It does not however cut down on LOC, so I'm somewhat on the fence.
I think that's somehow also around code maintenance (not only LOC), say for example
if we want to add more "child_type" in the check (no need to remember to update both
locations).
I didn't include checking the child_type in that function since it is
unrelated to instr_time, so it sadly wouldn't help with that. We could
macro-ize the child_type check were we to add another child_type.
- Melanie
Hi,
On 2025-02-27 06:50:41 +0000, Bertrand Drouvot wrote:
On Wed, Feb 26, 2025 at 01:45:39PM -0500, Melanie Plageman wrote:
Thanks for the continued review!
On Wed, Feb 26, 2025 at 2:41 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Wed, Feb 26, 2025 at 01:46:19PM +0900, Fujii Masao wrote:
With the current patch, when log_connections is enabled, the connection time is always
captured, and which might introduce performance overhead. No? Some users who enable
log_connections may not want this extra detail and want to avoid such overhead.
So, would it make sense to extend log_connections with a new option like "timing" and
log the connection time only when "timing" is specified?+1, I also think it's a good idea to let users decide if they want the timing
measurement overhead (and it's common practice with track_io_timing,
track_wal_io_timing, the newly track_cost_delay_timing for example)It seems to me like the extra timing collected and the one additional
log message isn't enough overhead to justify its own guc (for now).Agree. IIUC, I think that Fujii-san's idea was to extend log_connections with
a new option "timing" (i.e move it from ConfigureNamesBool to say
ConfigureNamesEnum with say on, off and timing?). I think that's a good idea.
I don't think the timing overhead is a relevant factor here - compared to the
fork of a new connection or performing authentication the cost of taking a few
timestamps is neglegible. A timestamp costs 10s to 100s of cycles, a fork many
many millions. Even if you have a really slow timestamp function, it's still
going to be way way cheaper.
Greetings,
Andres Freund
Hi,
On 2025-02-27 11:08:04 -0500, Melanie Plageman wrote:
I was just talking to Andres off-list and he mentioned that the volume
of log_connections messages added in recent releases can really be a
problem for users.
For some added color: I've seen plenty systems where almost all the log volume
is log_connection messages, which they *have* to enable for various regulatory
reasons. It'd still be a lot if we just emitted one message for each
connection, but logging three (and possibly four with $thread), for each
connection makes it substantially worse.
He said ideally we would emit one message which consolidated these (and make
sure we did so for failed connections too detailing the successfully
completed stages).
A combined message would also not *quite* replace all use-cases, e.g. if you
want to debug arriving connections or auth problems you do want the additional
messages. But yea, for normal operation, I do think most folks want just one
message.
However, since that is a bigger project (with more refactoring, etc),
he suggested that we change log_connections to a GUC_LIST
(ConfigureNamesString) with options like "none", "received,
"authenticated", "authorized", "all".
Yep.
Then we could add one like "established" for the final message and
timings my patch set adds. I think the overhead of an additional log
message being emitted probably outweighs the overhead of taking those
additional timings.
To bikeshed a bit: "established" could be the TCP connection establishment
just as well. I'd go for "completed" or "timings".
String GUCs are a lot more work than enum GUCs, so I was thinking if
there is a way to do it as an enum.I think we want the user to be able to specify a list of all the log
messages they want included, not just have each one include the
previous ones. So, then it probably has to be a list right? There is
no good design that would fit as an enum.
I don't see a way to comfortably shove this into an enum either.
Greetings,
Andres Freund
On Thu, Feb 27, 2025 at 11:30 AM Andres Freund <andres@anarazel.de> wrote:
On 2025-02-27 11:08:04 -0500, Melanie Plageman wrote:
However, since that is a bigger project (with more refactoring, etc),
he suggested that we change log_connections to a GUC_LIST
(ConfigureNamesString) with options like "none", "received,
"authenticated", "authorized", "all".Yep.
I've done a draft of this in attached v7 (see 0001). It still needs
polishing (e.g. I need to figure out where to put the new guc hook
functions and enums and such), but I want to see if this is a viable
direction forward.
I'm worried the list of possible connection log messages could get
unwieldy if we add many more.
- Melanie
Attachments:
v7-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v7-0002-Add-connection-establishment-duration-logging.patchDownload
From 6f7f1a2e3ea660596bee0348691e192b8c2fce24 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 27 Feb 2025 17:33:38 -0500
Subject: [PATCH v7 2/2] Add connection establishment duration logging
Add durations for several key parts of connection establishment when
log_connections is enabled.
For a new incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment time with logging.
To make this portable, the timestamps captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 2 +-
src/backend/postmaster/launch_backend.c | 23 +++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 12 +++++++++++-
src/backend/tcop/postgres.c | 21 +++++++++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 12 ++++++++++++
src/include/libpq/libpq-be.h | 2 ++
src/include/miscadmin.h | 2 ++
src/include/portability/instr_time.h | 9 +++++++++
src/include/postmaster/postmaster.h | 8 ++++++++
src/include/tcop/backend_startup.h | 5 +++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 97 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c927efd22d1..82b06179a38 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7324,7 +7324,7 @@ local0.* /var/log/postgresql
<para>
Causes each attempted connection to the server to be logged, as well as
successful completion of both client authentication (if necessary) and
- authorization.
+ authorization. Also logs connection establishment component durations.
</para>
<para>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..3fe68601899 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,11 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_time);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +245,15 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Calculate total fork duration in child backend for logging */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -618,6 +632,15 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
+ /* Calculate total fork duration in child backend for logging */
+ if ((log_connections & LOG_CONNECTION_READY) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
+ }
+
/* Close the postmaster's sockets (as soon as we know them) */
ClosePostmasterPorts(child_type == B_LOGGER);
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index aef63793aa7..71aa47c8d36 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1536,7 +1536,7 @@ check_log_connections(char **newval, void **extra, GucSource source)
if (pg_strcasecmp(*newval, "none") == 0)
effval = "";
else if (pg_strcasecmp(*newval, "all") == 0)
- effval = "received, authenticated, authorized";
+ effval = "received, authenticated, authorized, timings";
/* Need a modifiable copy of string */
rawstring = pstrdup(effval);
@@ -1558,6 +1558,8 @@ check_log_connections(char **newval, void **extra, GucSource source)
flags |= LOG_CONNECTION_AUTHENTICATED;
else if (pg_strcasecmp(item, "authorized") == 0)
flags |= LOG_CONNECTION_AUTHORIZED;
+ else if (pg_strcasecmp(item, "timings") == 0)
+ flags |= LOG_CONNECTION_TIMINGS;
else if (pg_strcasecmp(item, "none") == 0 || pg_strcasecmp(item, "all") == 0)
{
GUC_check_errdetail("Cannot specify \"%s\" in a list of other log_connections options.", item);
@@ -1759,7 +1761,14 @@ ServerLoop(void)
ClientSocket s;
if (AcceptConnection(events[i].fd, &s) == STATUS_OK)
+ {
+ /*
+ * Capture time that Postmaster got a socket from accept
+ * (for logging connection establishment duration)
+ */
+ INSTR_TIME_SET_CURRENT(s.creation_time);
BackendStartup(&s);
+ }
/* We no longer need the open socket in this process */
if (s.sock != PGINVALID_SOCKET)
@@ -3585,6 +3594,7 @@ BackendStartup(ClientSocket *client_sock)
/* Pass down canAcceptConnections state */
startup_data.canAcceptConnections = cac;
+ INSTR_TIME_SET_ZERO(startup_data.fork_time);
bn->rw = NULL;
/* Hasn't asked to be notified about any bgworkers yet */
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..d753c6bbd5b 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_TIMINGS) &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(MyClientSocket->creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 8811f2ba3e6..44e086ceca7 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,16 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +258,11 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..b16ad69efff 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
@@ -252,6 +253,7 @@ typedef struct ClientSocket
{
pgsocket sock; /* File descriptor */
SockAddr raddr; /* remote addr (client) */
+ instr_time creation_time;
} ClientSocket;
#ifdef USE_SSL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index adc693a6b2b..dae7551e7ad 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -127,6 +128,12 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+typedef struct ConnectionTiming
+{
+ instr_time fork_duration;
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
@@ -149,6 +156,7 @@ typedef enum ConnectionLogType
LOG_CONNECTION_RECEIVED,
LOG_CONNECTION_AUTHENTICATED,
LOG_CONNECTION_AUTHORIZED,
+ LOG_CONNECTION_TIMINGS,
} ConnectionLogType;
#endif /* _POSTMASTER_H */
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..7d9c43ce77b 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,9 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /* Time at which the postmaster initiates a fork of a backend process */
+ instr_time fork_time;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 59839013d28..ea7163fdada 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -485,6 +485,7 @@ ConnStatusType
ConnType
ConnectionLogType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
v7-0001-Make-log_connections-a-list.patchtext/x-patch; charset=US-ASCII; name=v7-0001-Make-log_connections-a-list.patchDownload
From 6178535e0be8e25e4209f42d52c8402b6d1e4724 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 27 Feb 2025 17:32:17 -0500
Subject: [PATCH v7 1/2] Make log_connections a list
Instead of a boolean, make log_connections a list of the possible log
connections events to log. all and none mimic current behavior of on and
off. This is in response to feedback that log_connections logs too many
messages.
---
doc/src/sgml/config.sgml | 25 ++++--
src/backend/libpq/auth.c | 5 +-
src/backend/postmaster/postmaster.c | 76 ++++++++++++++++++-
src/backend/tcop/backend_startup.c | 2 +-
src/backend/utils/init/postinit.c | 2 +-
src/backend/utils/misc/guc_tables.c | 20 ++---
src/backend/utils/misc/postgresql.conf.sample | 4 +-
src/include/postmaster/postmaster.h | 10 ++-
src/include/utils/guc_hooks.h | 2 +
.../libpq/t/005_negotiate_encryption.pl | 2 +-
src/test/authentication/t/001_password.pl | 2 +-
src/test/authentication/t/003_peer.pl | 2 +-
src/test/authentication/t/005_sspi.pl | 2 +-
src/test/kerberos/t/001_auth.pl | 2 +-
src/test/ldap/t/001_auth.pl | 2 +-
src/test/ldap/t/002_bindpasswd.pl | 2 +-
.../t/001_mutated_bindpasswd.pl | 2 +-
.../modules/oauth_validator/t/001_server.pl | 2 +-
.../modules/oauth_validator/t/002_client.pl | 2 +-
.../postmaster/t/002_connection_limits.pl | 2 +-
src/test/postmaster/t/003_start_stop.pl | 2 +-
src/test/recovery/t/013_crash_restart.pl | 2 +-
src/test/recovery/t/022_crash_temp_files.pl | 2 +-
src/test/recovery/t/032_relfilenode_reuse.pl | 2 +-
src/test/recovery/t/037_invalid_database.pl | 2 +-
src/test/ssl/t/SSL/Server.pm | 2 +-
src/tools/ci/pg_ci_base.conf | 2 +-
src/tools/pgindent/typedefs.list | 1 +
28 files changed, 141 insertions(+), 42 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e55700f35b8..c927efd22d1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -140,7 +140,7 @@
An example of what this file might look like is:
<programlisting>
# This is a comment
-log_connections = yes
+log_connections = 'all'
log_destination = 'syslog'
search_path = '"$user", public'
shared_buffers = 128MB
@@ -337,7 +337,7 @@ UPDATE pg_settings SET setting = reset_val WHERE name = 'configuration_parameter
<option>-c name=value</option> command-line parameter, or its equivalent
<option>--name=value</option> variation. For example,
<programlisting>
-postgres -c log_connections=yes --log-destination='syslog'
+postgres -c log_connections='all' --log-destination='syslog'
</programlisting>
Settings provided in this way override those set via
<filename>postgresql.conf</filename> or <command>ALTER SYSTEM</command>,
@@ -7315,20 +7315,31 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes each attempted connection to the server to be logged, as well as
+ successful completion of both client authentication (if necessary) and
+ authorization.
+ </para>
+
+ <para>
+ May be set to <literal>none</literal> to disable this logging,
+ <literal>all</literal> to enable all connection log message types, or a
+ comma-separated list of the specific connection log message types to
+ emit. The valid options are <literal>received</literal>,
+ <literal>authenticated</literal>, and <literal>authorized</literal>.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
+ The default is <literal>'none'</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..b4cc8cb4dd1 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -349,7 +349,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +633,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATED) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..aef63793aa7 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -118,6 +118,7 @@
#include "utils/pidfile.h"
#include "utils/timestamp.h"
#include "utils/varlena.h"
+#include "utils/guc_hooks.h"
#ifdef EXEC_BACKEND
#include "common/file_utils.h"
@@ -237,7 +238,8 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
+char *log_connections_string = NULL;
+int log_connections = 0;
bool enable_bonjour = false;
char *bonjour_name;
@@ -1517,6 +1519,78 @@ checkControlFile(void)
FreeFile(fp);
}
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ int flags = 0;
+ char *rawstring = NULL;
+ List *elemlist;
+ ListCell *l;
+
+ char *effval = *newval;
+
+ /*
+ * log_connections can be "all" or "none" or a list of comma separated
+ * options
+ */
+ if (pg_strcasecmp(*newval, "none") == 0)
+ effval = "";
+ else if (pg_strcasecmp(*newval, "all") == 0)
+ effval = "received, authenticated, authorized";
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(effval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"%s\".",
+ "log_connections");
+ goto error;
+ }
+
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "received") == 0)
+ flags |= LOG_CONNECTION_RECEIVED;
+ else if (pg_strcasecmp(item, "authenticated") == 0)
+ flags |= LOG_CONNECTION_AUTHENTICATED;
+ else if (pg_strcasecmp(item, "authorized") == 0)
+ flags |= LOG_CONNECTION_AUTHORIZED;
+ else if (pg_strcasecmp(item, "none") == 0 || pg_strcasecmp(item, "all") == 0)
+ {
+ GUC_check_errdetail("Cannot specify \"%s\" in a list of other log_connections options.", item);
+ goto error;
+ }
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ goto error;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+
+ /* Save the flags in *extra, for use by assign_log_connections */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+
+error:
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+}
+
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
+
/*
* Determine how long should we let ServerLoop sleep, in milliseconds.
*
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..2c208c25356 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -202,7 +202,7 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
/* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_RECEIVED)
{
if (remote_port[0])
ereport(LOG,
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..8811f2ba3e6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -251,7 +251,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..7dfa11810a8 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4850,6 +4841,17 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about successful connection."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "none",
+ check_log_connections, assign_log_connections, NULL
+ },
+
{
{"synchronized_standby_slots", PGC_SIGHUP, REPLICATION_PRIMARY,
gettext_noop("Lists streaming replication standby server replication slot "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5362ff80519..4f54719dd9e 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections='all'". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,7 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = 'none'
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..adc693a6b2b 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,8 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
+extern PGDLLIMPORT int log_connections;
+extern PGDLLIMPORT char *log_connections_string;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
@@ -143,4 +144,11 @@ typedef enum DispatchOption
extern DispatchOption parse_dispatch_option(const char *name);
+typedef enum ConnectionLogType
+{
+ LOG_CONNECTION_RECEIVED,
+ LOG_CONNECTION_AUTHENTICATED,
+ LOG_CONNECTION_AUTHORIZED,
+} ConnectionLogType;
+
#endif /* _POSTMASTER_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/interfaces/libpq/t/005_negotiate_encryption.pl b/src/interfaces/libpq/t/005_negotiate_encryption.pl
index c834fa5149a..cb56e09d50b 100644
--- a/src/interfaces/libpq/t/005_negotiate_encryption.pl
+++ b/src/interfaces/libpq/t/005_negotiate_encryption.pl
@@ -107,7 +107,7 @@ $node->append_conf(
listen_addresses = '$hostaddr'
# Capturing the EVENTS that occur during tests requires these settings
-log_connections = on
+log_connections = 'all'
log_disconnections = on
trace_connection_negotiation = on
lc_messages = 'C'
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 4ce22ccbdf2..72c187fbdc0 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -63,7 +63,7 @@ sub test_conn
# Initialize primary node
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
# could fail in FIPS mode
diff --git a/src/test/authentication/t/003_peer.pl b/src/test/authentication/t/003_peer.pl
index 69ba73bd2b9..8e1b0477d6b 100644
--- a/src/test/authentication/t/003_peer.pl
+++ b/src/test/authentication/t/003_peer.pl
@@ -71,7 +71,7 @@ sub test_role
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
# Set pg_hba.conf with the peer authentication.
diff --git a/src/test/authentication/t/005_sspi.pl b/src/test/authentication/t/005_sspi.pl
index b480b702590..f9b61babcd7 100644
--- a/src/test/authentication/t/005_sspi.pl
+++ b/src/test/authentication/t/005_sspi.pl
@@ -18,7 +18,7 @@ if (!$windows_os || $use_unix_sockets)
# Initialize primary node
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
my $huge_pages_status =
diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl
index 6748b109dec..afc3130e720 100644
--- a/src/test/kerberos/t/001_auth.pl
+++ b/src/test/kerberos/t/001_auth.pl
@@ -65,7 +65,7 @@ $node->append_conf(
'postgresql.conf', qq{
listen_addresses = '$hostaddr'
krb_server_keyfile = '$krb->{keytab}'
-log_connections = on
+log_connections = 'all'
lc_messages = 'C'
});
$node->start;
diff --git a/src/test/ldap/t/001_auth.pl b/src/test/ldap/t/001_auth.pl
index 352b0fc1fa7..b6ee386679d 100644
--- a/src/test/ldap/t/001_auth.pl
+++ b/src/test/ldap/t/001_auth.pl
@@ -47,7 +47,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
$node->safe_psql('postgres', 'CREATE USER test0;');
diff --git a/src/test/ldap/t/002_bindpasswd.pl b/src/test/ldap/t/002_bindpasswd.pl
index f8beba2b279..2acf96aeb6d 100644
--- a/src/test/ldap/t/002_bindpasswd.pl
+++ b/src/test/ldap/t/002_bindpasswd.pl
@@ -43,7 +43,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
$node->safe_psql('postgres', 'CREATE USER test0;');
diff --git a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
index 9b062e1c800..4591b5568a8 100644
--- a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
+++ b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
@@ -42,7 +42,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"shared_preload_libraries = 'ldap_password_func'");
$node->start;
diff --git a/src/test/modules/oauth_validator/t/001_server.pl b/src/test/modules/oauth_validator/t/001_server.pl
index 6fa59fbeb25..27829680439 100644
--- a/src/test/modules/oauth_validator/t/001_server.pl
+++ b/src/test/modules/oauth_validator/t/001_server.pl
@@ -43,7 +43,7 @@ if ($ENV{with_python} ne 'yes')
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
$node->start;
diff --git a/src/test/modules/oauth_validator/t/002_client.pl b/src/test/modules/oauth_validator/t/002_client.pl
index ab83258d736..aa944ba5f2f 100644
--- a/src/test/modules/oauth_validator/t/002_client.pl
+++ b/src/test/modules/oauth_validator/t/002_client.pl
@@ -26,7 +26,7 @@ if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ /\boauth\b/)
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
$node->start;
diff --git a/src/test/postmaster/t/002_connection_limits.pl b/src/test/postmaster/t/002_connection_limits.pl
index 8cfa6e0ced5..a221501c206 100644
--- a/src/test/postmaster/t/002_connection_limits.pl
+++ b/src/test/postmaster/t/002_connection_limits.pl
@@ -19,7 +19,7 @@ $node->init(
$node->append_conf('postgresql.conf', "max_connections = 6");
$node->append_conf('postgresql.conf', "reserved_connections = 2");
$node->append_conf('postgresql.conf', "superuser_reserved_connections = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'all'");
$node->start;
$node->safe_psql(
diff --git a/src/test/postmaster/t/003_start_stop.pl b/src/test/postmaster/t/003_start_stop.pl
index 036b296f72b..08cd7a4857e 100644
--- a/src/test/postmaster/t/003_start_stop.pl
+++ b/src/test/postmaster/t/003_start_stop.pl
@@ -29,7 +29,7 @@ $node->append_conf('postgresql.conf', "max_connections = 5");
$node->append_conf('postgresql.conf', "max_wal_senders = 0");
$node->append_conf('postgresql.conf', "autovacuum_max_workers = 1");
$node->append_conf('postgresql.conf', "max_worker_processes = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'all'");
$node->append_conf('postgresql.conf', "log_min_messages = debug2");
$node->append_conf('postgresql.conf',
"authentication_timeout = '$authentication_timeout s'");
diff --git a/src/test/recovery/t/013_crash_restart.pl b/src/test/recovery/t/013_crash_restart.pl
index cd848918d00..2b6ec94cd75 100644
--- a/src/test/recovery/t/013_crash_restart.pl
+++ b/src/test/recovery/t/013_crash_restart.pl
@@ -27,7 +27,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET restart_after_crash = 1;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = 'all';
SELECT pg_reload_conf();]);
# Run psql, keeping session alive, so we have an alive backend to kill.
diff --git a/src/test/recovery/t/022_crash_temp_files.pl b/src/test/recovery/t/022_crash_temp_files.pl
index 483a416723f..5d4b695c193 100644
--- a/src/test/recovery/t/022_crash_temp_files.pl
+++ b/src/test/recovery/t/022_crash_temp_files.pl
@@ -26,7 +26,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET remove_temp_files_after_crash = on;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = 'all';
ALTER SYSTEM SET work_mem = '64kB';
ALTER SYSTEM SET restart_after_crash = on;
SELECT pg_reload_conf();]);
diff --git a/src/test/recovery/t/032_relfilenode_reuse.pl b/src/test/recovery/t/032_relfilenode_reuse.pl
index ddb7223b337..388a50add7a 100644
--- a/src/test/recovery/t/032_relfilenode_reuse.pl
+++ b/src/test/recovery/t/032_relfilenode_reuse.pl
@@ -14,7 +14,7 @@ $node_primary->init(allows_streaming => 1);
$node_primary->append_conf(
'postgresql.conf', q[
allow_in_place_tablespaces = true
-log_connections=on
+log_connections='all'
# to avoid "repairing" corruption
full_page_writes=off
log_min_messages=debug2
diff --git a/src/test/recovery/t/037_invalid_database.pl b/src/test/recovery/t/037_invalid_database.pl
index bdf39397397..ea62650b767 100644
--- a/src/test/recovery/t/037_invalid_database.pl
+++ b/src/test/recovery/t/037_invalid_database.pl
@@ -15,7 +15,7 @@ $node->append_conf(
autovacuum = off
max_prepared_transactions=5
log_min_duration_statement=0
-log_connections=on
+log_connections='all'
log_disconnections=on
));
diff --git a/src/test/ssl/t/SSL/Server.pm b/src/test/ssl/t/SSL/Server.pm
index 447469d8937..7bc45ee7611 100644
--- a/src/test/ssl/t/SSL/Server.pm
+++ b/src/test/ssl/t/SSL/Server.pm
@@ -200,7 +200,7 @@ sub configure_test_server_for_ssl
$node->append_conf(
'postgresql.conf', <<EOF
fsync=off
-log_connections=on
+log_connections='all'
log_hostname=on
listen_addresses='$serverhost'
log_statement=all
diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf
index d8faa9c26c1..5593827eb4f 100644
--- a/src/tools/ci/pg_ci_base.conf
+++ b/src/tools/ci/pg_ci_base.conf
@@ -8,7 +8,7 @@ max_prepared_transactions = 10
# Settings that make logs more useful
log_autovacuum_min_duration = 0
log_checkpoints = true
-log_connections = true
+log_connections = 'all'
log_disconnections = true
log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
log_lock_waits = true
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfbab589d61..59839013d28 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -483,6 +483,7 @@ ConnCacheKey
ConnParams
ConnStatusType
ConnType
+ConnectionLogType
ConnectionStateEnum
ConsiderSplitContext
Const
--
2.34.1
Hi,
On Thu, Feb 27, 2025 at 11:14:56AM -0500, Andres Freund wrote:
I don't think the timing overhead is a relevant factor here - compared to the
fork of a new connection or performing authentication the cost of taking a few
timestamps is neglegible. A timestamp costs 10s to 100s of cycles, a fork many
many millions. Even if you have a really slow timestamp function, it's still
going to be way way cheaper.
That's a very good point, it has to be put in perspective. The difference in
scale is so significant that the timing collection shouldn't be a concern.
Fair point!
Now I'm thinking what about "if" the connection was on a multi-threaded model?
I think we could reach the same conclusion as thread creation overhead is
still substantial (allocating stack space, initializing thread state, and other
kernel-level operations) as compare to a really slow timestamp function.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Thu, Feb 27, 2025 at 11:08:04AM -0500, Melanie Plageman wrote:
I was just talking to Andres off-list and he mentioned that the volume
of log_connections messages added in recent releases can really be a
problem for users. He said ideally we would emit one message which
consolidated these (and make sure we did so for failed connections too
detailing the successfully completed stages).However, since that is a bigger project (with more refactoring, etc),
he suggested that we change log_connections to a GUC_LIST
(ConfigureNamesString) with options like "none", "received,
"authenticated", "authorized", "all".Then we could add one like "established" for the final message and
timings my patch set adds. I think the overhead of an additional log
message being emitted probably outweighs the overhead of taking those
additional timings.String GUCs are a lot more work than enum GUCs, so I was thinking if
there is a way to do it as an enum.I think we want the user to be able to specify a list of all the log
messages they want included, not just have each one include the
previous ones. So, then it probably has to be a list right? There is
no good design that would fit as an enum.
Interesting idea... Yeah, that would sound weird with an enum. I could think
about providing an enum per possible combination but I think that would
generate things like 2^N enum and won't be really user friendly (also that would
double each time we'd want to add a new possible "value" becoming quickly
unmanageable).
So yeah, I can't think of anything better than GUC_LIST.
I think that's somehow also around code maintenance (not only LOC), say for example
if we want to add more "child_type" in the check (no need to remember to update both
locations).I didn't include checking the child_type in that function since it is
unrelated to instr_time, so it sadly wouldn't help with that. We could
macro-ize the child_type check were we to add another child_type.
Yup but my idea was to put all those line:
"
if (Log_connections &&
(child_type == B_BACKEND || child_type == B_WAL_SENDER))
{
instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;
conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
}
"
into a dedicated helper function.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Thu, Feb 27, 2025 at 05:55:19PM -0500, Melanie Plageman wrote:
On Thu, Feb 27, 2025 at 11:30 AM Andres Freund <andres@anarazel.de> wrote:
On 2025-02-27 11:08:04 -0500, Melanie Plageman wrote:
However, since that is a bigger project (with more refactoring, etc),
he suggested that we change log_connections to a GUC_LIST
(ConfigureNamesString) with options like "none", "received,
"authenticated", "authorized", "all".Yep.
I've done a draft of this in attached v7 (see 0001).
Thanks for the patch!
It still needs polishing (e.g. I need to figure out where to put the new guc hook
functions and enums and such)
yeah, I wonder if it would make sense to create new dedicated "connection_logging"
file(s).
, but I want to see if this is a viable direction forward.
I'm worried the list of possible connection log messages could get
unwieldy if we add many more.
Thinking out loud, I'm not sure we'd add "many more" but maybe what we could do
is to introduce some predefined groups like:
'basic' (that would be equivalent to 'received, + timings introduced in 0002')
'security' (equivalent to 'authenticated,authorized')
Not saying we need to create this kind of predefined groups now, just an idea
in case we'd add many more: that could "help" the user experience, or are you
more concerned about the code maintenance?
Just did a quick test, (did not look closely at the code), and it looks like
that:
'all': does not report the "received" (but does report authenticated and authorized)
'received': does not work?
'authenticated': works
'authorized': works
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Fri, Feb 28, 2025 at 12:16 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
Yup but my idea was to put all those line:
"
if (Log_connections &&
(child_type == B_BACKEND || child_type == B_WAL_SENDER))
{
instr_time fork_time = ((BackendStartupData *) startup_data)->fork_time;conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(fork_time);
}
"into a dedicated helper function.
I ended up finding a bug that means that that exact code isn't
duplicated twice now. For EXEC_BACKEND, I have to wait to calculate
the duration until after reading the GUC values but I need to get the
fork end time before that.
I tried coming up with an inline helper to replace this and most
things I tried felt awkward. It has to have backend type and start
time as parameters. And all of these places we have to be super
careful that the GUCs have already been read before using
log_connections, so it seems a bit unsafe to check log_connections
(the global variable) in a function. Someone could call the function
in a different place and maybe not know that log_connections won't be
set there.
Also, I also wasn't sure if it would be weird to call a function like
"LogConnectionTiming()" which in many cases doesn't log the connection
timing (because it is a different backend type).
But maybe I'm not thinking about it correctly. What were you imagining?
- Melanie
On Fri, Feb 28, 2025 at 12:54 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Thu, Feb 27, 2025 at 05:55:19PM -0500, Melanie Plageman wrote:
It still needs polishing (e.g. I need to figure out where to put the new guc hook
functions and enums and such)yeah, I wonder if it would make sense to create new dedicated "connection_logging"
file(s).
I think for now there isn't enough for a new file. I think these are
probably okay in postmaster.c since this has to do with postmaster
starting new connections.
I'm worried the list of possible connection log messages could get
unwieldy if we add many more.Thinking out loud, I'm not sure we'd add "many more" but maybe what we could do
is to introduce some predefined groups like:'basic' (that would be equivalent to 'received, + timings introduced in 0002')
'security' (equivalent to 'authenticated,authorized')Not saying we need to create this kind of predefined groups now, just an idea
in case we'd add many more: that could "help" the user experience, or are you
more concerned about the code maintenance?
I was more worried about the user having to provide very long lists.
But groupings could be a sensible solution in the future.
Just did a quick test, (did not look closely at the code), and it looks like
that:'all': does not report the "received" (but does report authenticated and authorized)
'received': does not work?
'authenticated': works
'authorized': works
Thanks for testing! Ouch, there were many bugs in that prototype. My
enums were broken and a few other things.
I found some bugs in 0002 as well. Attached v8 fixes the issues I found so far.
We have to be really careful about when log_connections is actually
set before we use it -- I fixed some issues with that.
I decided to move the creation_time out of ClientSocket and into
BackendStartupData, but that meant I had to save it in the
ConnectionTiming because we don't have access to the
BackendStartupData anymore in PostgresMain(). That means
ConnectionTiming is now a mixture of times and durations. I think that
should be okay (i.e. not too confusing and gross), but I'm not sure.
It is called ConnectionTiming and not ConnectionDuration, after all.
- Melanie
Attachments:
v8-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v8-0001-Modularize-log_connections-output.patchDownload
From 0ee1cd306a4c27e5ec0d1860ce7cd219bb761fed Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 27 Feb 2025 17:32:17 -0500
Subject: [PATCH v8 1/2] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC of the
connection stages to log. This obsoletes log_disconnections, as
'disconnected' is now a log_connections option.
The current log_connections options are 'received', 'authenticated',
'authorized', and 'disconnected'. The empty string disables all
connection logging. 'all' enables all available connection logging.
This gives users more control over the volume of connection logging.
This patch has the same behavior as [1] but was developed independently.
The idea to replace log_disconnections with a log_connections option
does, however, come from that thread.
[1] https://www.postgresql.org/message-id/flat/CAA8Fd-qCB96uwfgMKrzfNs32mqqysi53yZFNVaRNJ6xDthZEgA%40mail.gmail.com
Discussion: https://postgr.es/m/flat/Z8FPpJPDLHWNuV2c%40ip-10-97-1-34.eu-west-3.compute.internal#35525b4425c5f8ecfe52ec6ad859ef9a
---
doc/src/sgml/config.sgml | 48 +++++-------
src/backend/libpq/auth.c | 8 +-
src/backend/postmaster/postmaster.c | 77 ++++++++++++++++++-
src/backend/tcop/backend_startup.c | 4 +-
src/backend/tcop/postgres.c | 13 +---
src/backend/utils/init/postinit.c | 2 +-
src/backend/utils/misc/guc_tables.c | 29 +++----
src/backend/utils/misc/postgresql.conf.sample | 5 +-
src/include/postmaster/postmaster.h | 16 +++-
src/include/tcop/tcopprot.h | 1 -
src/include/utils/guc_hooks.h | 2 +
.../libpq/t/005_negotiate_encryption.pl | 3 +-
src/test/authentication/t/001_password.pl | 2 +-
src/test/authentication/t/003_peer.pl | 2 +-
src/test/authentication/t/005_sspi.pl | 2 +-
src/test/kerberos/t/001_auth.pl | 2 +-
src/test/ldap/t/001_auth.pl | 2 +-
src/test/ldap/t/002_bindpasswd.pl | 2 +-
.../t/001_mutated_bindpasswd.pl | 2 +-
.../modules/oauth_validator/t/001_server.pl | 2 +-
.../modules/oauth_validator/t/002_client.pl | 2 +-
.../postmaster/t/002_connection_limits.pl | 2 +-
src/test/postmaster/t/003_start_stop.pl | 2 +-
src/test/recovery/t/013_crash_restart.pl | 2 +-
src/test/recovery/t/022_crash_temp_files.pl | 2 +-
src/test/recovery/t/032_relfilenode_reuse.pl | 2 +-
src/test/recovery/t/037_invalid_database.pl | 3 +-
src/test/ssl/t/SSL/Server.pm | 2 +-
src/tools/ci/pg_ci_base.conf | 3 +-
src/tools/pgindent/typedefs.list | 1 +
30 files changed, 155 insertions(+), 90 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e55700f35b8..4d2bbb3f720 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -140,7 +140,7 @@
An example of what this file might look like is:
<programlisting>
# This is a comment
-log_connections = yes
+log_connections = 'all'
log_destination = 'syslog'
search_path = '"$user", public'
shared_buffers = 128MB
@@ -337,7 +337,7 @@ UPDATE pg_settings SET setting = reset_val WHERE name = 'configuration_parameter
<option>-c name=value</option> command-line parameter, or its equivalent
<option>--name=value</option> variation. For example,
<programlisting>
-postgres -c log_connections=yes --log-destination='syslog'
+postgres -c log_connections='all' --log-destination='syslog'
</programlisting>
Settings provided in this way override those set via
<filename>postgresql.conf</filename> or <command>ALTER SYSTEM</command>,
@@ -7315,20 +7315,28 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
- Only superusers and users with the appropriate <literal>SET</literal>
- privilege can change this parameter at session start,
- and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
+ Causes the specified stages of each connection attempt to the server to
+ be logged. Only superusers and users with the appropriate
+ <literal>SET</literal> privilege can change this parameter at session
+ start, and it cannot be changed at all within a session. The default is
+ <literal>''</literal> which causes no connection logging messages to be
+ emitted.
+ </para>
+
+ <para>
+ May be set to <literal>''</literal> to disable connection logging,
+ <literal>'all'</literal> to log all available connection stages, or a
+ comma-separated list of specific connection stages to log. The valid
+ options are <literal>received</literal>,
+ <literal>authenticated</literal>, <literal>authorized</literal>, and
+ <literal>disconnected</literal>.
</para>
<note>
@@ -7342,26 +7350,6 @@ local0.* /var/log/postgresql
</listitem>
</varlistentry>
- <varlistentry id="guc-log-disconnections" xreflabel="log_disconnections">
- <term><varname>log_disconnections</varname> (<type>boolean</type>)
- <indexterm>
- <primary><varname>log_disconnections</varname> configuration parameter</primary>
- </indexterm>
- </term>
- <listitem>
- <para>
- Causes session terminations to be logged. The log output
- provides information similar to <varname>log_connections</varname>,
- plus the duration of the session.
- Only superusers and users with the appropriate <literal>SET</literal>
- privilege can change this parameter at session start,
- and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
- </para>
- </listitem>
- </varlistentry>
-
-
<varlistentry id="guc-log-duration" xreflabel="log_duration">
<term><varname>log_duration</varname> (<type>boolean</type>)
<indexterm>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..da7b46720d4 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -317,7 +317,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authenticated' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +350,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +634,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATED) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..315f5a55a34 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -118,6 +118,7 @@
#include "utils/pidfile.h"
#include "utils/timestamp.h"
#include "utils/varlena.h"
+#include "utils/guc_hooks.h"
#ifdef EXEC_BACKEND
#include "common/file_utils.h"
@@ -237,7 +238,8 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
+char *log_connections_string = NULL;
+int log_connections = 0;
bool enable_bonjour = false;
char *bonjour_name;
@@ -1517,6 +1519,79 @@ checkControlFile(void)
FreeFile(fp);
}
+/*
+ * Validate the input for the log_connections GUC.
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ int flags = 0;
+ char *rawstring = NULL;
+ List *elemlist;
+ ListCell *l;
+
+ char *effval = *newval;
+
+ /*
+ * log_connections can be 'all', '', or a list of comma-separated options.
+ */
+ if (pg_strcasecmp(*newval, "all") == 0)
+ effval = "received, authenticated, authorized, disconnected";
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(effval);
+
+ if (!SplitGUCList(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ goto log_connections_error;
+ }
+
+ foreach(l, elemlist)
+ {
+ char *item = (char *) lfirst(l);
+
+ if (pg_strcasecmp(item, "received") == 0)
+ flags |= LOG_CONNECTION_RECEIVED;
+ else if (pg_strcasecmp(item, "authenticated") == 0)
+ flags |= LOG_CONNECTION_AUTHENTICATED;
+ else if (pg_strcasecmp(item, "authorized") == 0)
+ flags |= LOG_CONNECTION_AUTHORIZED;
+ else if (pg_strcasecmp(item, "disconnected") == 0)
+ flags |= LOG_CONNECTION_DISCONNECTED;
+ else if (pg_strcasecmp(item, "all") == 0)
+ {
+ GUC_check_errdetail("Must specify \"all\" alone with no additional options, whitespace, or characters.");
+ goto log_connections_error;
+ }
+ else
+ {
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ goto log_connections_error;
+ }
+ }
+
+ pfree(rawstring);
+ list_free(elemlist);
+
+ /* Save the flags in *extra, for use by assign_log_connections */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+
+log_connections_error:
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+}
+
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
+
/*
* Determine how long should we let ServerLoop sleep, in milliseconds.
*
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..887c2f177ef 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -201,8 +201,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIVED)
{
if (remote_port[0])
ereport(LOG,
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..8b403a6cc3d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -89,9 +89,6 @@ const char *debug_query_string; /* client-supplied query string */
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
-/* flag for logging end of session */
-bool Log_disconnections = false;
-
int log_statement = LOGSTMT_NONE;
/* wait N seconds to allow attach from a debugger */
@@ -3653,10 +3650,7 @@ set_debug_options(int debug_flag, GucContext context, GucSource source)
SetConfigOption("log_min_messages", "notice", context, source);
if (debug_flag >= 1 && context == PGC_POSTMASTER)
- {
- SetConfigOption("log_connections", "true", context, source);
- SetConfigOption("log_disconnections", "true", context, source);
- }
+ SetConfigOption("log_connections", "all", context, source);
if (debug_flag >= 2)
SetConfigOption("log_statement", "all", context, source);
if (debug_flag >= 3)
@@ -4271,9 +4265,10 @@ PostgresMain(const char *dbname, const char *username)
/*
* Also set up handler to log session end; we have to wait till now to be
- * sure Log_disconnections has its final value.
+ * sure log_connections has its final value.
*/
- if (IsUnderPostmaster && Log_disconnections)
+ if (IsUnderPostmaster &&
+ (log_connections & LOG_CONNECTION_DISCONNECTED))
on_proc_exit(log_disconnections, 0);
pgstat_report_connect(MyDatabaseId);
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 318600d6d02..8811f2ba3e6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -251,7 +251,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..5b73270c8a9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -1238,15 +1229,6 @@ struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
- {
- {"log_disconnections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs end of a session, including duration."),
- NULL
- },
- &Log_disconnections,
- false,
- NULL, NULL, NULL
- },
{
{"log_replication_commands", PGC_SUSET, LOGGING_WHAT,
gettext_noop("Logs each replication command."),
@@ -4850,6 +4832,17 @@ struct config_string ConfigureNamesString[] =
check_debug_io_direct, assign_debug_io_direct, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about events during connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
{
{"synchronized_standby_slots", PGC_SIGHUP, REPLICATION_PRIMARY,
gettext_noop("Lists streaming replication standby server replication slot "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5362ff80519..bf611317b3f 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections='all'". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,8 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
-#log_disconnections = off
+#log_connections = ''
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
#log_hostname = off
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..ffa30e94aff 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,8 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
+extern PGDLLIMPORT int log_connections;
+extern PGDLLIMPORT char *log_connections_string;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
@@ -143,4 +144,17 @@ typedef enum DispatchOption
extern DispatchOption parse_dispatch_option(const char *name);
+/*
+ * These flags correspond to the various stages of connection establishment
+ * which may be logged. check_log_connections() converts the options specified
+ * in the log_connections GUC to some combination of these values.
+ */
+typedef enum ConnectionLogOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_DISCONNECTED = (1 << 3),
+} ConnectionLogOption;
+
#endif /* _POSTMASTER_H */
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index a62367f7793..dde9b7e9b59 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -36,7 +36,6 @@ typedef enum
LOGSTMT_ALL, /* log all statements */
} LogStmtLevel;
-extern PGDLLIMPORT bool Log_disconnections;
extern PGDLLIMPORT int log_statement;
/* Flags for restrict_nonsystem_relation_kind value */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/interfaces/libpq/t/005_negotiate_encryption.pl b/src/interfaces/libpq/t/005_negotiate_encryption.pl
index c834fa5149a..d488aa1f1e2 100644
--- a/src/interfaces/libpq/t/005_negotiate_encryption.pl
+++ b/src/interfaces/libpq/t/005_negotiate_encryption.pl
@@ -107,8 +107,7 @@ $node->append_conf(
listen_addresses = '$hostaddr'
# Capturing the EVENTS that occur during tests requires these settings
-log_connections = on
-log_disconnections = on
+log_connections = 'all'
trace_connection_negotiation = on
lc_messages = 'C'
});
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 4ce22ccbdf2..72c187fbdc0 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -63,7 +63,7 @@ sub test_conn
# Initialize primary node
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
# could fail in FIPS mode
diff --git a/src/test/authentication/t/003_peer.pl b/src/test/authentication/t/003_peer.pl
index 69ba73bd2b9..8e1b0477d6b 100644
--- a/src/test/authentication/t/003_peer.pl
+++ b/src/test/authentication/t/003_peer.pl
@@ -71,7 +71,7 @@ sub test_role
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
# Set pg_hba.conf with the peer authentication.
diff --git a/src/test/authentication/t/005_sspi.pl b/src/test/authentication/t/005_sspi.pl
index b480b702590..f9b61babcd7 100644
--- a/src/test/authentication/t/005_sspi.pl
+++ b/src/test/authentication/t/005_sspi.pl
@@ -18,7 +18,7 @@ if (!$windows_os || $use_unix_sockets)
# Initialize primary node
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
my $huge_pages_status =
diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl
index 6748b109dec..afc3130e720 100644
--- a/src/test/kerberos/t/001_auth.pl
+++ b/src/test/kerberos/t/001_auth.pl
@@ -65,7 +65,7 @@ $node->append_conf(
'postgresql.conf', qq{
listen_addresses = '$hostaddr'
krb_server_keyfile = '$krb->{keytab}'
-log_connections = on
+log_connections = 'all'
lc_messages = 'C'
});
$node->start;
diff --git a/src/test/ldap/t/001_auth.pl b/src/test/ldap/t/001_auth.pl
index 352b0fc1fa7..b6ee386679d 100644
--- a/src/test/ldap/t/001_auth.pl
+++ b/src/test/ldap/t/001_auth.pl
@@ -47,7 +47,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
$node->safe_psql('postgres', 'CREATE USER test0;');
diff --git a/src/test/ldap/t/002_bindpasswd.pl b/src/test/ldap/t/002_bindpasswd.pl
index f8beba2b279..2acf96aeb6d 100644
--- a/src/test/ldap/t/002_bindpasswd.pl
+++ b/src/test/ldap/t/002_bindpasswd.pl
@@ -43,7 +43,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->start;
$node->safe_psql('postgres', 'CREATE USER test0;');
diff --git a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
index 9b062e1c800..4591b5568a8 100644
--- a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
+++ b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
@@ -42,7 +42,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"shared_preload_libraries = 'ldap_password_func'");
$node->start;
diff --git a/src/test/modules/oauth_validator/t/001_server.pl b/src/test/modules/oauth_validator/t/001_server.pl
index 6fa59fbeb25..27829680439 100644
--- a/src/test/modules/oauth_validator/t/001_server.pl
+++ b/src/test/modules/oauth_validator/t/001_server.pl
@@ -43,7 +43,7 @@ if ($ENV{with_python} ne 'yes')
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
$node->start;
diff --git a/src/test/modules/oauth_validator/t/002_client.pl b/src/test/modules/oauth_validator/t/002_client.pl
index ab83258d736..aa944ba5f2f 100644
--- a/src/test/modules/oauth_validator/t/002_client.pl
+++ b/src/test/modules/oauth_validator/t/002_client.pl
@@ -26,7 +26,7 @@ if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ /\boauth\b/)
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'all'\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
$node->start;
diff --git a/src/test/postmaster/t/002_connection_limits.pl b/src/test/postmaster/t/002_connection_limits.pl
index 8cfa6e0ced5..a221501c206 100644
--- a/src/test/postmaster/t/002_connection_limits.pl
+++ b/src/test/postmaster/t/002_connection_limits.pl
@@ -19,7 +19,7 @@ $node->init(
$node->append_conf('postgresql.conf', "max_connections = 6");
$node->append_conf('postgresql.conf', "reserved_connections = 2");
$node->append_conf('postgresql.conf', "superuser_reserved_connections = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'all'");
$node->start;
$node->safe_psql(
diff --git a/src/test/postmaster/t/003_start_stop.pl b/src/test/postmaster/t/003_start_stop.pl
index 036b296f72b..08cd7a4857e 100644
--- a/src/test/postmaster/t/003_start_stop.pl
+++ b/src/test/postmaster/t/003_start_stop.pl
@@ -29,7 +29,7 @@ $node->append_conf('postgresql.conf', "max_connections = 5");
$node->append_conf('postgresql.conf', "max_wal_senders = 0");
$node->append_conf('postgresql.conf', "autovacuum_max_workers = 1");
$node->append_conf('postgresql.conf', "max_worker_processes = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'all'");
$node->append_conf('postgresql.conf', "log_min_messages = debug2");
$node->append_conf('postgresql.conf',
"authentication_timeout = '$authentication_timeout s'");
diff --git a/src/test/recovery/t/013_crash_restart.pl b/src/test/recovery/t/013_crash_restart.pl
index cd848918d00..2b6ec94cd75 100644
--- a/src/test/recovery/t/013_crash_restart.pl
+++ b/src/test/recovery/t/013_crash_restart.pl
@@ -27,7 +27,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET restart_after_crash = 1;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = 'all';
SELECT pg_reload_conf();]);
# Run psql, keeping session alive, so we have an alive backend to kill.
diff --git a/src/test/recovery/t/022_crash_temp_files.pl b/src/test/recovery/t/022_crash_temp_files.pl
index 483a416723f..5d4b695c193 100644
--- a/src/test/recovery/t/022_crash_temp_files.pl
+++ b/src/test/recovery/t/022_crash_temp_files.pl
@@ -26,7 +26,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET remove_temp_files_after_crash = on;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = 'all';
ALTER SYSTEM SET work_mem = '64kB';
ALTER SYSTEM SET restart_after_crash = on;
SELECT pg_reload_conf();]);
diff --git a/src/test/recovery/t/032_relfilenode_reuse.pl b/src/test/recovery/t/032_relfilenode_reuse.pl
index ddb7223b337..388a50add7a 100644
--- a/src/test/recovery/t/032_relfilenode_reuse.pl
+++ b/src/test/recovery/t/032_relfilenode_reuse.pl
@@ -14,7 +14,7 @@ $node_primary->init(allows_streaming => 1);
$node_primary->append_conf(
'postgresql.conf', q[
allow_in_place_tablespaces = true
-log_connections=on
+log_connections='all'
# to avoid "repairing" corruption
full_page_writes=off
log_min_messages=debug2
diff --git a/src/test/recovery/t/037_invalid_database.pl b/src/test/recovery/t/037_invalid_database.pl
index bdf39397397..e03b963f917 100644
--- a/src/test/recovery/t/037_invalid_database.pl
+++ b/src/test/recovery/t/037_invalid_database.pl
@@ -15,8 +15,7 @@ $node->append_conf(
autovacuum = off
max_prepared_transactions=5
log_min_duration_statement=0
-log_connections=on
-log_disconnections=on
+log_connections='all'
));
$node->start;
diff --git a/src/test/ssl/t/SSL/Server.pm b/src/test/ssl/t/SSL/Server.pm
index 447469d8937..7bc45ee7611 100644
--- a/src/test/ssl/t/SSL/Server.pm
+++ b/src/test/ssl/t/SSL/Server.pm
@@ -200,7 +200,7 @@ sub configure_test_server_for_ssl
$node->append_conf(
'postgresql.conf', <<EOF
fsync=off
-log_connections=on
+log_connections='all'
log_hostname=on
listen_addresses='$serverhost'
log_statement=all
diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf
index d8faa9c26c1..b05a4844b26 100644
--- a/src/tools/ci/pg_ci_base.conf
+++ b/src/tools/ci/pg_ci_base.conf
@@ -8,7 +8,6 @@ max_prepared_transactions = 10
# Settings that make logs more useful
log_autovacuum_min_duration = 0
log_checkpoints = true
-log_connections = true
-log_disconnections = true
+log_connections = 'all'
log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
log_lock_waits = true
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 56989aa0b84..d155dcff036 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -483,6 +483,7 @@ ConnCacheKey
ConnParams
ConnStatusType
ConnType
+ConnectionLogOption
ConnectionStateEnum
ConsiderSplitContext
Const
--
2.34.1
v8-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v8-0002-Add-connection-establishment-duration-logging.patchDownload
From 22584b57cda98058d2a1d714cc8d45c676935d9d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 27 Feb 2025 17:33:38 -0500
Subject: [PATCH v8 2/2] Add connection establishment duration logging
Add log_connections option 'timings' which logs durations for several
key parts of connection establishment when specified.
For a new incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment and initialization time with logging.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the ClientSocket
and BackendStartupData structures instead of simply saved in backend
local memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 15 +++++----
src/backend/postmaster/launch_backend.c | 45 +++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 10 +++++-
src/backend/tcop/postgres.c | 21 ++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 10 ++++++
src/include/libpq/libpq-be.h | 1 +
src/include/miscadmin.h | 2 ++
src/include/portability/instr_time.h | 9 +++++
src/include/postmaster/postmaster.h | 23 +++++++++++++
src/include/tcop/backend_startup.h | 14 ++++++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 146 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4d2bbb3f720..b1f38b99ee0 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7331,12 +7331,15 @@ local0.* /var/log/postgresql
</para>
<para>
- May be set to <literal>''</literal> to disable connection logging,
- <literal>'all'</literal> to log all available connection stages, or a
- comma-separated list of specific connection stages to log. The valid
- options are <literal>received</literal>,
- <literal>authenticated</literal>, <literal>authorized</literal>, and
- <literal>disconnected</literal>.
+ May be set to <literal>''</literal> to disable this logging,
+ <literal>'all'</literal> to enable all connection log message types, or
+ a comma-separated list of the specific connection log message types to
+ emit. The valid options are <literal>received</literal>,
+ <literal>authenticated</literal>, <literal>authorized</literal>,
+ <literal>disconnected</literal>, and <literal>timings</literal>. When
+ <literal>timings</literal> is specified, a message is logged the first
+ time the connection is ready for query containing connection
+ establishment durations.
</para>
<note>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..67fdad55414 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,11 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_started);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +245,22 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /*
+ * Transfer over any log_connections 'timings' data we need that was
+ * collected by the postmaster. We save the time the socket was
+ * created to later log the total connection establishment and setup
+ * time. Calculate the total fork duration now since we have the start
+ * and end times.
+ */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(data->fork_started);
+ conn_timing.creation_time = data->socket_created;
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,6 +607,7 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ instr_time fork_ended;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
@@ -598,6 +620,13 @@ SubPostmasterMain(int argc, char *argv[])
if (argc != 3)
elog(FATAL, "invalid subpostmaster invocation");
+ /*
+ * Set the fork end time. We haven't read the GUC file yet, so we don't
+ * know if we will use this value for logging, but getting the time is
+ * relatively little overhead.
+ */
+ INSTR_TIME_SET_CURRENT(fork_ended);
+
/* Find the entry in child_process_kinds */
if (strncmp(argv[1], "--forkchild=", 12) != 0)
elog(FATAL, "invalid subpostmaster invocation (--forkchild argument missing)");
@@ -648,6 +677,22 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /*
+ * Transfer over any log_connections 'timings' data we need that was
+ * collected by the postmaster. We save the time the socket was created to
+ * later log the total connection establishment and setup time. Calculate
+ * the total fork duration now since we have the start and end times.
+ */
+ if ((log_connections & LOG_CONNECTION_TIMINGS) &&
+ (child_type == B_BACKEND || child_type == B_WAL_SENDER))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ conn_timing.fork_duration = fork_ended;
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration, data->fork_started);
+ conn_timing.creation_time = data->socket_created;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 315f5a55a34..67d388ff2d5 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -1536,7 +1536,7 @@ check_log_connections(char **newval, void **extra, GucSource source)
* log_connections can be 'all', '', or a list of comma-separated options.
*/
if (pg_strcasecmp(*newval, "all") == 0)
- effval = "received, authenticated, authorized, disconnected";
+ effval = "received, authenticated, authorized, disconnected, timings";
/* Need a modifiable copy of string */
rawstring = pstrdup(effval);
@@ -1559,6 +1559,8 @@ check_log_connections(char **newval, void **extra, GucSource source)
flags |= LOG_CONNECTION_AUTHORIZED;
else if (pg_strcasecmp(item, "disconnected") == 0)
flags |= LOG_CONNECTION_DISCONNECTED;
+ else if (pg_strcasecmp(item, "timings") == 0)
+ flags |= LOG_CONNECTION_TIMINGS;
else if (pg_strcasecmp(item, "all") == 0)
{
GUC_check_errdetail("Must specify \"all\" alone with no additional options, whitespace, or characters.");
@@ -3553,6 +3555,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment duration).
+ */
+ INSTR_TIME_SET_CURRENT(startup_data.socket_created);
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8b403a6cc3d..8c6fdf4bc9d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4147,6 +4147,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4602,6 +4603,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_TIMINGS) &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(conn_timing.creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 8811f2ba3e6..6d39acffb27 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -190,9 +190,15 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ if (log_connections & LOG_CONNECTION_TIMINGS)
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -251,6 +257,10 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ if (log_connections & LOG_CONNECTION_TIMINGS)
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 7fe92b15477..7579d96563e 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -58,6 +58,7 @@ typedef struct
#include "datatype/timestamp.h"
#include "libpq/hba.h"
#include "libpq/pqcomm.h"
+#include "portability/instr_time.h"
/*
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index ffa30e94aff..70818e70d2a 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -127,6 +128,27 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders. Used for
+ * log_connections 'timings' option.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created. This is used to log the
+ * total connection establishment and setup time from accept() to first
+ * being ready for query.
+ */
+ instr_time creation_time;
+
+ /* How long it took the backend to be forked. */
+ instr_time fork_duration;
+
+ /* How long authentication took */
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
@@ -155,6 +177,7 @@ typedef enum ConnectionLogOption
LOG_CONNECTION_AUTHENTICATED = (1 << 1),
LOG_CONNECTION_AUTHORIZED = (1 << 2),
LOG_CONNECTION_DISCONNECTED = (1 << 3),
+ LOG_CONNECTION_TIMINGS = (1 << 4),
} ConnectionLogOption;
#endif /* _POSTMASTER_H */
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..03192e21387 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
@@ -37,6 +39,18 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the postmaster initiates a fork of a backend process Only
+ * used for client and wal sender connections.
+ */
+ instr_time fork_started;
+
+ /*
+ * Time at which the connection client socket is created Only used for
+ * client and wal sender connections.
+ */
+ instr_time socket_created;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d155dcff036..c030047a107 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -485,6 +485,7 @@ ConnStatusType
ConnType
ConnectionLogOption
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
Hi,
On Fri, Feb 28, 2025 at 05:42:37PM -0500, Melanie Plageman wrote:
And all of these places we have to be super
careful that the GUCs have already been read before using
log_connections, so it seems a bit unsafe to check log_connections
(the global variable) in a function.
Yeah, that could be checking before calling the function (the caller would be
responsible for checking log_connections)
Also, I also wasn't sure if it would be weird to call a function like
"LogConnectionTiming()" which in many cases doesn't log the connection
timing (because it is a different backend type).But maybe I'm not thinking about it correctly. What were you imagining?
I did not imagine that much ;-) I was just seeing this code being duplicated
and just thought about to avoid the duplication. But now that I read your comments
above then I think we could just macro-ize the child_type check (as you mentioned
up-thread). That would avoid the risk to forget to update the 3 locations doing the
exact same check should we add a new child type in the game.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Fri, Feb 28, 2025 at 05:52:35PM -0500, Melanie Plageman wrote:
On Fri, Feb 28, 2025 at 12:54 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Thu, Feb 27, 2025 at 05:55:19PM -0500, Melanie Plageman wrote:
It still needs polishing (e.g. I need to figure out where to put the new guc hook
functions and enums and such)yeah, I wonder if it would make sense to create new dedicated "connection_logging"
file(s).I think for now there isn't enough for a new file. I think these are
probably okay in postmaster.c since this has to do with postmaster
starting new connections.
Agree.
I'm worried the list of possible connection log messages could get
unwieldy if we add many more.Thinking out loud, I'm not sure we'd add "many more" but maybe what we could do
is to introduce some predefined groups like:'basic' (that would be equivalent to 'received, + timings introduced in 0002')
'security' (equivalent to 'authenticated,authorized')Not saying we need to create this kind of predefined groups now, just an idea
in case we'd add many more: that could "help" the user experience, or are you
more concerned about the code maintenance?I was more worried about the user having to provide very long lists.
But groupings could be a sensible solution in the future.
Okay, yeah most probably.
Just did a quick test, (did not look closely at the code), and it looks like
that:'all': does not report the "received" (but does report authenticated and authorized)
'received': does not work?
'authenticated': works
'authorized': worksThanks for testing! Ouch, there were many bugs in that prototype. My
enums were broken and a few other things.I found some bugs in 0002 as well. Attached v8 fixes the issues I found so far.
Thanks for the updated version!
We have to be really careful about when log_connections is actually
set before we use it -- I fixed some issues with that.
Good catch! Yeah for the EXEC_BACKEND case we need to wait that read_nondefault_variables()
in SubPostmasterMain() is executed.
Looking at 0001, a few random comments:
=== 1
#include "utils/varlena.h"
+#include "utils/guc_hooks.h"
wrong ordering?
=== 2
+/*
+ * Validate the input for the log_connections GUC.
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me. I just have a few
following comments about it.
I was just wondering why:
"
+ else if (pg_strcasecmp(item, "all") == 0)
+ {
+ GUC_check_errdetail("Must specify \"all\" alone with no additional options, whitespace, or characters.");
+ goto log_connections_error;
+ }
"
but yeah that probably makes more sense that way.
=== 3
+ if (pg_strcasecmp(*newval, "all") == 0)
+ effval = "received, authenticated, authorized, disconnected";
Not sure this one is needed, see comment "=== 5".
=== 4
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
Not written in the exact same way as assign_debug_io_direct() but that's fine.
=== 5
+typedef enum ConnectionLogOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_DISCONNECTED = (1 << 3),
+} ConnectionLogOption;
I wonder if it would make sense to add LOG_CONNECTION_ALL here
(LOG_CONNECTION_RECEIVED | LOG_CONNECTION_AUTHENTICATED ..)
That sounds a better place (than defining "all" in check_log_connections()) to
me. It's also how it is done in MonotonicFunction.
That way we could avoid the special case in check_log_connections() and it
looks less likely that we miss to add new flags in LOG_CONNECTION_ALL should
we add more options, thoughts?
We'd need to change all the checks that you added (around log_connections) to
also add a check on LOG_CONNECTION_ALL though.
=== 6
Also not sure if it's worth adding a "MAX" (like it's done for relopt_kind)
because we'd probably not go over it anyay.
"
/* some compilers treat enums as signed ints, so we can't use 1 << 31 */
RELOPT_KIND_MAX = (1 << 30)
"
Just mentioning in passing as I just realized there is this check in relopt_kind.
=== 7
s/ConnectionLogOption/LogConnectionOption/? (as it is linked to "log_connections")
=== 8
All the TAP tests have been modified that way:
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'all'");
Is it worh to add some checks for the other values?
I did a few manual checks and that seems to work as expected.
I'll look at 0002 later.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On 2025/03/04 1:14, Bertrand Drouvot wrote:
=== 2
+/* + * Validate the input for the log_connections GUC. + */ +bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me.
I guess that check_log_connections() is implemented to use SplitGUCList()
because check_debug_io_direct() does too. However, according to the comment for
SplitGUCList() — "This is used to split the value of a GUC_LIST_QUOTE GUC variable",
it seems more appropriate to use SplitIdentifierString() instead,
like other options such as log_destination, since log_connections is not
a GUC_LIST_QUOTE GUC.
" + else if (pg_strcasecmp(item, "all") == 0) + { + GUC_check_errdetail("Must specify \"all\" alone with no additional options, whitespace, or characters."); + goto log_connections_error; + } "but yeah that probably makes more sense that way.
Wouldn't this restriction be a bit confusing for users? If there is no particular
reason for it, wouldn’t it be simpler and clearer to allow "all" with additional
options, whitespace, or characters?
=== 5
+typedef enum ConnectionLogOption +{ + LOG_CONNECTION_RECEIVED = (1 << 0), + LOG_CONNECTION_AUTHENTICATED = (1 << 1), + LOG_CONNECTION_AUTHORIZED = (1 << 2), + LOG_CONNECTION_DISCONNECTED = (1 << 3), +} ConnectionLogOption;I wonder if it would make sense to add LOG_CONNECTION_ALL here
(LOG_CONNECTION_RECEIVED | LOG_CONNECTION_AUTHENTICATED ..)
+1
Here are some review comments from me:
+ <literal>''</literal> which causes no connection logging messages to be
+ emitted.
<snip>
+ May be set to <literal>''</literal> to disable connection logging,
Wouldn't it be clearer to describe this as "the empty string <literal>''</literal>"
like other GUC options, so users immediately understand what it represents?
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about events during connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
Like log_destination, wouldn’t it be better to explicitly list
all valid values in the long description?
+#log_connections = ''
It would also be helpful to include all valid values in a comment
within postgresql.conf.sample, making it easier for users to configure.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On Mon, Mar 3, 2025 at 11:14 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Fri, Feb 28, 2025 at 05:52:35PM -0500, Melanie Plageman wrote:
+bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me. I just have a few
following comments about it.
So, I started thinking more about this patch. I wonder if what we
really want to do is add a new GUC type that is somewhere between a
list and an enum. What I'm imagining is basically a set -- you could
specify parameters in a comma separated list like before but without
quotes:
SET log_connections = received,authorized;
It would be for GUCs that can be any combination of a set of
predetermined values.
There are actually a few list gucs that would benefit from this
structure: see log_destination, debug_io_direct,
restrict_nonsystem_relation_kind. And I imagine there are others.
It would reduce code duplication for those (see the boiler plate
string parsing and handling for all of these check functions). If this
was implemented, log_connections wouldn't even need its own check
function (I think).
It also might be easier to do tab completion (depending on how it is
implemented).
And I think it would be even simpler to do your idea of groups and to
have aliases for different combinations of options. And it would let
us keep 'on' and 'off' as backwards compatibility aliases.
Maybe if I tried implementing it, it wouldn't work at all. But, as it
stands, I'm feeling nervous about making two of the most common GUCs
in the world (log_connections and log_disconnections) less structured.
And maybe, instead of 'all', we want '*'? I think using the idea of a
set may open us up to better interface ideas.
All that being said, this is hardly a project for March 3rd. So, in
service of moving 0002 forward, I think I want to walk back to an idea
everyone else on this thread was keen on: making log_connections an
enum with 'on', 'off', and 'timings'. This moves us in the direction
of having more granular control in log_connections without breaking
any existing users and paving the way for a project like what I
outlined above. What do you think?
I was just wondering why:
" + else if (pg_strcasecmp(item, "all") == 0) + { + GUC_check_errdetail("Must specify \"all\" alone with no additional options, whitespace, or characters."); + goto log_connections_error; + } "but yeah that probably makes more sense that way.
Well, you can't have other punctuation or trailing commas with other
list GUCs. The whitespace I could add code to support. But I think
this goes back to my feelings about why this whole endeavor might be
sketchy.
- Melanie
On 2025/03/01 7:52, Melanie Plageman wrote:
On Fri, Feb 28, 2025 at 12:54 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Thu, Feb 27, 2025 at 05:55:19PM -0500, Melanie Plageman wrote:
It still needs polishing (e.g. I need to figure out where to put the new guc hook
functions and enums and such)yeah, I wonder if it would make sense to create new dedicated "connection_logging"
file(s).I think for now there isn't enough for a new file. I think these are
probably okay in postmaster.c since this has to do with postmaster
starting new connections.I'm worried the list of possible connection log messages could get
unwieldy if we add many more.Thinking out loud, I'm not sure we'd add "many more" but maybe what we could do
is to introduce some predefined groups like:'basic' (that would be equivalent to 'received, + timings introduced in 0002')
'security' (equivalent to 'authenticated,authorized')Not saying we need to create this kind of predefined groups now, just an idea
in case we'd add many more: that could "help" the user experience, or are you
more concerned about the code maintenance?I was more worried about the user having to provide very long lists.
But groupings could be a sensible solution in the future.Just did a quick test, (did not look closely at the code), and it looks like
that:'all': does not report the "received" (but does report authenticated and authorized)
'received': does not work?
'authenticated': works
'authorized': worksThanks for testing! Ouch, there were many bugs in that prototype. My
enums were broken and a few other things.I found some bugs in 0002 as well. Attached v8 fixes the issues I found so far.
We have to be really careful about when log_connections is actually
set before we use it -- I fixed some issues with that.
When log_connection is empty string in postgresql.conf and I ran
"psql -d "options=-clog_connections=all"", I got the following log message.
You can see the total duration in this message is unexpected.
It looks like this happens because creation_time wasn’t collected,
as log_connections was empty before the fork.
LOG: connection establishment times (ms): total: 1148052469, fork: 0, authentication: 0
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Hi,
On Mon, Mar 03, 2025 at 06:24:59PM -0500, Melanie Plageman wrote:
On Mon, Mar 3, 2025 at 11:14 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Fri, Feb 28, 2025 at 05:52:35PM -0500, Melanie Plageman wrote:
+bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me. I just have a few
following comments about it.So, I started thinking more about this patch. I wonder if what we
really want to do is add a new GUC type that is somewhere between a
list and an enum. What I'm imagining is basically a set -- you could
specify parameters in a comma separated list like before but without
quotes:SET log_connections = received,authorized;
It would be for GUCs that can be any combination of a set of
predetermined values.
There are actually a few list gucs that would benefit from this
structure: see log_destination, debug_io_direct,
restrict_nonsystem_relation_kind. And I imagine there are others.It would reduce code duplication for those (see the boiler plate
string parsing and handling for all of these check functions). If this
was implemented, log_connections wouldn't even need its own check
function (I think).It also might be easier to do tab completion (depending on how it is
implemented).And I think it would be even simpler to do your idea of groups and to
have aliases for different combinations of options. And it would let
us keep 'on' and 'off' as backwards compatibility aliases.
hm, I think that's a very interesting idea. Being able to preserve backward
compatibility is certainly a big bonus (as long as the values keep doing the
exact same thing).
Maybe if I tried implementing it, it wouldn't work at all. But, as it
stands, I'm feeling nervous about making two of the most common GUCs
in the world (log_connections and log_disconnections) less structured.
I think the patch was in a good shape but yeah that might be too late in
the release cycle for such a change... And even if we were to merge it that
would not be a user friendly thing to change it in 2 releases in a row (I'm
imagining 19 implementing your new "SET" idea).
All that being said, this is hardly a project for March 3rd.
Fully agree.
So, in
service of moving 0002 forward, I think I want to walk back to an idea
everyone else on this thread was keen on: making log_connections an
enum with 'on', 'off', and 'timings'. This moves us in the direction
of having more granular control in log_connections without breaking
any existing users and paving the way for a project like what I
outlined above. What do you think?
I do agree (see above as to why). That said, it means that 18 will not "solve"
the concerns about the volume of log_connections messages that you and Andres
shared up-thread.
But given that:
- 9afffcb833d3 and e48b19c5db31 are the most recent ones that added extra
messages (so added in pre-18)
- the new 'timings' option will help to prevent to add extra message in 18 (
if not desired)
Then it's not like we will add a lot of extra messages by default on 18.
So being on the safe side of things and postpone this to 19 seems a perfectly
reasonable thing to do.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Tue, Mar 4, 2025 at 4:40 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Mon, Mar 03, 2025 at 06:24:59PM -0500, Melanie Plageman wrote:
And I think it would be even simpler to do your idea of groups and to
have aliases for different combinations of options. And it would let
us keep 'on' and 'off' as backwards compatibility aliases.hm, I think that's a very interesting idea. Being able to preserve backward
compatibility is certainly a big bonus (as long as the values keep doing the
exact same thing).Maybe if I tried implementing it, it wouldn't work at all. But, as it
stands, I'm feeling nervous about making two of the most common GUCs
in the world (log_connections and log_disconnections) less structured.
I tried to mock up a prototype this morning of PGC_ENUM_LIST (or
PGC_SET, not sure what to call it) and actually parsing the enum list
input was easy to do, but there are a bunch of other things to figure
out that will take time.
Anyway, I'll start polishing the enum "on", "off", "timings" version
of my patch.
- Melanie
On Mon, Mar 3, 2025 at 9:07 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
I did not imagine that much ;-) I was just seeing this code being duplicated
and just thought about to avoid the duplication. But now that I read your comments
above then I think we could just macro-ize the child_type check (as you mentioned
up-thread). That would avoid the risk to forget to update the 3 locations doing the
exact same check should we add a new child type in the game.
Is there a word we could use to describe what B_BACKEND and
B_WAL_SENDER have in common? They are the only backend types that will
go through the kind of external connection establishment steps (I
think), but I don't know a very accurate way to make that distinction
(which is required to come up with a useful macro name).
- Melanie
Attached v9 implements log_connections as an enum with a new third
value "timings".
On Mon, Mar 3, 2025 at 11:14 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Fri, Feb 28, 2025 at 05:52:35PM -0500, Melanie Plageman wrote:
We have to be really careful about when log_connections is actually
set before we use it -- I fixed some issues with that.Good catch! Yeah for the EXEC_BACKEND case we need to wait that read_nondefault_variables()
in SubPostmasterMain() is executed.
Due to this and the bug Fujii-san reported with passing options to
psql (process_startup_options() is called too late for us to use the
value of log_connections as a gate for getting any of the timings),
I've decided to get the time regardless of the setting of
log_connections. This means we can be sure to have valid values at the
end when the message needs to be emitted.
I was wondering if we should remove the backend type condition
(B_WAL_SENDER/B_BACKEND) too. It is only guarding capturing the fork
start and end time. That's just two measurements. Maybe we should just
always get fork start and end time. Spending extra time in forking due
to excess CPU activity (or some other reason) would be relevant for
all backend types, not just wal sender and client backends. We do only
want to log it in those cases, though...
+bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me. I just have a few
following comments about it.
Yep, one more reason I think PGC_SET is a good idea. Code deduplication.
+typedef enum ConnectionLogOption +{ + LOG_CONNECTION_RECEIVED = (1 << 0), + LOG_CONNECTION_AUTHENTICATED = (1 << 1), + LOG_CONNECTION_AUTHORIZED = (1 << 2), + LOG_CONNECTION_DISCONNECTED = (1 << 3), +} ConnectionLogOption;I wonder if it would make sense to add LOG_CONNECTION_ALL here
(LOG_CONNECTION_RECEIVED | LOG_CONNECTION_AUTHENTICATED ..)That sounds a better place (than defining "all" in check_log_connections()) to
me. It's also how it is done in MonotonicFunction.
The attached patch doesn't work exactly the same because it is a
regular enum and not a string list (so no "all"), but I do still have
a LogConnectionOption. The user interface is that timings includes all
the same messages as "on", however for the enum, I've defined it like
this:
typedef enum LogConnectionOption
{
LOG_CONNECTION_BASIC = (1 << 0),
LOG_CONNECTION_TIMINGS = (1 << 1),
} LogConnectionOption;
and then in the enum array, I have this
static const struct config_enum_entry log_connections_options[] = {
{"timings", LOG_CONNECTION_TIMINGS | LOG_CONNECTION_BASIC, false},
This is in contrast to MonotonicFunction which defines
MONOTONICFUNC_BOTH as including both values in the enum
typedef enum MonotonicFunction
{
MONOTONICFUNC_NONE = 0,
MONOTONICFUNC_INCREASING = (1 << 0),
MONOTONICFUNC_DECREASING = (1 << 1),
MONOTONICFUNC_BOTH = MONOTONICFUNC_INCREASING | MONOTONICFUNC_DECREASING,
} MonotonicFunction;
I think the way I've done it makes sense because the "business logic"
of how "timings" includes all the messages from "on" doesn't have to
pollute the code and doesn't take away our ability to use the enum
values independently.
When I want to emit a regular logging message, I can just check
if (log_connections & LOG_CONNECTION_BASIC)
which will work for "timings" too, but without taking away the
flexibility to use the enum values discretely.
Also not sure if it's worth adding a "MAX" (like it's done for relopt_kind)
because we'd probably not go over it anyay."
/* some compilers treat enums as signed ints, so we can't use 1 << 31 */
RELOPT_KIND_MAX = (1 << 30)
That actually seems kind of broken -- what if it is on a platform
where integers are 16 bits? Maybe we don't support any of those.
Honestly I find the MAX more confusing than not having it at all. I
think we have to assume that programmers know or can look up how many
bits they can use of an enum in C. Or we can just upgrade to c23 and
have explicit underlying types for enums! :)
s/ConnectionLogOption/LogConnectionOption/? (as it is linked to "log_connections")
Yep. Good idea. I've changed this.
All the TAP tests have been modified that way:
-$node->append_conf('postgresql.conf', "log_connections = on"); +$node->append_conf('postgresql.conf', "log_connections = 'all'");Is it worh to add some checks for the other values?
Now we are back to only having 3 values, but I was thinking if we
wanted to add some test exercising "timings". We can't test any
specific values of any of the durations emitted, so it would have to
just be a test that that log message was emitted. I don't see a very
good place to add such a test though. There is
src/test/postmaster/t/002_connection_limits.pl -- which has to do with
testing different connection limits (like max_connections). I don't
see any other TAP tests that are testing connections. There are a
whole bunch of auth tests, I suppose.
I did a few manual checks and that seems to work as expected.
Thanks so much for your continued feedback and review!
- Melanie
Attachments:
v9-0001-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v9-0001-Add-connection-establishment-duration-logging.patchDownload
From 5d6947f93d8126ca5b0a86d28a7cb2e57a252b0a Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 4 Mar 2025 09:44:34 -0500
Subject: [PATCH v9] Add connection establishment duration logging
Add log_connections option 'timings' which logs durations for several
key parts of connection establishment and setup when specified.
For a new incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. Provide visibility into
authentication and fork duration as well as the end-to-end connection
establishment and initialization time with logging.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData structures instead of simply saved in backend local
memory inherited by the child process.
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 17 +++++++----
src/backend/libpq/auth.c | 4 +--
src/backend/postmaster/launch_backend.c | 38 +++++++++++++++++++++++++
src/backend/postmaster/postmaster.c | 8 +++++-
src/backend/tcop/backend_startup.c | 4 +--
src/backend/tcop/postgres.c | 21 ++++++++++++++
src/backend/utils/init/globals.c | 2 ++
src/backend/utils/init/postinit.c | 10 ++++++-
src/backend/utils/misc/guc_tables.c | 33 +++++++++++++++------
src/include/miscadmin.h | 2 ++
src/include/portability/instr_time.h | 9 ++++++
src/include/postmaster/postmaster.h | 35 ++++++++++++++++++++++-
src/include/tcop/backend_startup.h | 12 ++++++++
src/tools/pgindent/typedefs.list | 2 ++
14 files changed, 176 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..219c9ff6c5d 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,27 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>enum</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes each attempted connection to the server to be logged, as well as
+ successful completion of both client authentication (if necessary) and
+ authorization. Valid values are <literal>off</literal> (the default),
+ <literal>on</literal>, and <literal>timings</literal>.
+ <literal>timings</literal> emits all the messages emitted when
+ <structfield>log_connections</structfield> is <literal>on</literal>
+ plus an additional message at the end of connection setup containing
+ durations of various stages of connection establishment and setup.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..041ef9ba2cf 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -349,7 +349,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_BASIC)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +633,7 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if (log_connections & LOG_CONNECTION_BASIC && status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..2c45cbeb6d5 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (child_type == B_BACKEND || child_type == B_WAL_SENDER)
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_started);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,21 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /*
+ * Transfer over any log_connections 'timings' data we need that was
+ * collected by the postmaster. We save the time the socket was
+ * created to later log the total connection establishment and setup
+ * time. Calculate the total fork duration now since we have the start
+ * and end times.
+ */
+ if (child_type == B_BACKEND || child_type == B_WAL_SENDER)
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(data->fork_started);
+ conn_timing.creation_time = data->socket_created;
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,6 +605,7 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ instr_time fork_ended;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
@@ -615,6 +635,9 @@ SubPostmasterMain(int argc, char *argv[])
if (!found)
elog(ERROR, "unknown child kind %s", child_kind);
+ if (child_type == B_BACKEND || child_type == B_WAL_SENDER)
+ INSTR_TIME_SET_CURRENT(fork_ended);
+
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
@@ -648,6 +671,21 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /*
+ * Transfer over any log_connections 'timings' data we need that was
+ * collected by the postmaster. We save the time the socket was created to
+ * later log the total connection establishment and setup time. Calculate
+ * the total fork duration now since we have the start and end times.
+ */
+ if (child_type == B_BACKEND || child_type == B_WAL_SENDER)
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ conn_timing.fork_duration = fork_ended;
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration, data->fork_started);
+ conn_timing.creation_time = data->socket_created;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..68bc5e1f49e 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,7 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
+int log_connections = 0;
bool enable_bonjour = false;
char *bonjour_name;
@@ -3478,6 +3478,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment duration).
+ */
+ INSTR_TIME_SET_CURRENT(startup_data.socket_created);
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..da7f80c9337 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -201,8 +201,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_BASIC)
{
if (remote_port[0])
ereport(LOG,
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f2f75aa0f88..7bdd1f708fa 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -4153,6 +4153,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4608,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ log_connections & LOG_CONNECTION_TIMINGS &&
+ (AmRegularBackendProcess() || AmWalSenderProcess()))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(conn_timing.creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..3c7b14dd57d 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -43,6 +43,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..ad1437bb286 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -191,9 +191,14 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -252,7 +257,10 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ /* Calculate authentication duration for logging */
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
+ if (log_connections & LOG_CONNECTION_BASIC)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..a063cd10597 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -476,6 +476,20 @@ static const struct config_enum_entry wal_compression_options[] = {
{NULL, 0, false}
};
+static const struct config_enum_entry log_connections_options[] = {
+ {"off", 0, false},
+ {"on", LOG_CONNECTION_BASIC, false},
+ {"true", LOG_CONNECTION_BASIC, true},
+ {"false", 0, true},
+ {"yes", LOG_CONNECTION_BASIC, true},
+ {"no", 0, true},
+ {"timings", LOG_CONNECTION_TIMINGS | LOG_CONNECTION_BASIC, false},
+ {"1", LOG_CONNECTION_BASIC, true},
+ {"0", 0, true},
+ {NULL, 0, false}
+};
+
+
/*
* Options for enum values stored in other modules
*/
@@ -1219,15 +1233,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -5299,6 +5304,16 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about events during connection establishment."),
+ NULL,
+ },
+ &log_connections,
+ 0, log_connections_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..9dd18a2c74f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..7ff3aa5179a 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -15,6 +15,7 @@
#include "lib/ilist.h"
#include "miscadmin.h"
+#include "portability/instr_time.h"
/*
* A struct representing an active postmaster child process. This is used
@@ -51,6 +52,17 @@ typedef struct
extern int num_pmchild_slots;
#endif
+
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_BASIC = (1 << 0),
+ LOG_CONNECTION_TIMINGS = (1 << 1),
+} LogConnectionOption;
+
+
/* GUC options */
extern PGDLLIMPORT bool EnableSSL;
extern PGDLLIMPORT int SuperuserReservedConnections;
@@ -63,7 +75,7 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
+extern PGDLLIMPORT int log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
@@ -126,6 +138,27 @@ extern PMChild *AllocDeadEndChild(void);
extern bool ReleasePostmasterChildSlot(PMChild *pmchild);
extern PMChild *FindPostmasterChildByPid(int pid);
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders. Used for
+ * log_connections 'timings' option.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created. This is used to log the
+ * total connection establishment and setup time from accept() to first
+ * being ready for query.
+ */
+ instr_time creation_time;
+
+ /* How long it took the backend to be forked. */
+ instr_time fork_duration;
+
+ /* How long authentication took */
+ instr_time auth_duration;
+} ConnectionTiming;
+
/*
* These values correspond to the special must-be-first options for dispatching
* to various subprograms. parse_dispatch_option() can be used to convert an
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..62b4746ee60 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -37,6 +37,18 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the postmaster initiates a fork of a backend process.
+ * Only used for client and wal sender connections.
+ */
+ instr_time fork_started;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ instr_time socket_created;
} BackendStartupData;
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..3da74999634 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
@@ -1555,6 +1556,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
Thanks so much for your review!
On Mon, Mar 3, 2025 at 12:08 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
+bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me.I guess that check_log_connections() is implemented to use SplitGUCList()
because check_debug_io_direct() does too. However, according to the comment for
SplitGUCList() — "This is used to split the value of a GUC_LIST_QUOTE GUC variable",
it seems more appropriate to use SplitIdentifierString() instead,
like other options such as log_destination, since log_connections is not
a GUC_LIST_QUOTE GUC.
Ah, this was a very good catch. It seems like actually debug_io_direct
should use that too. Now I'm not using the string list method anymore,
but thanks for catching this.
+#log_connections = ''
It would also be helpful to include all valid values in a comment
within postgresql.conf.sample, making it easier for users to configure.
I haven't updated postgresql.conf.sample in v9 proposed here [1]/messages/by-id/CAAKRu_aoerKAOYKkc7-JGq2bixrYTbViK_7EeLNhFUGoT_omFw@mail.gmail.com. But
now that log_connections has three options, do you think I should
specify all three? It used to only specify off, but probably because
it was obvious that "on" was the opposite of "off".
- Melanie
[1]: /messages/by-id/CAAKRu_aoerKAOYKkc7-JGq2bixrYTbViK_7EeLNhFUGoT_omFw@mail.gmail.com
On Mon, Mar 3, 2025 at 11:45 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
When log_connection is empty string in postgresql.conf and I ran
"psql -d "options=-clog_connections=all"", I got the following log message.
You can see the total duration in this message is unexpected.
It looks like this happens because creation_time wasn’t collected,
as log_connections was empty before the fork.LOG: connection establishment times (ms): total: 1148052469, fork: 0, authentication: 0
Wow, yes, thanks for catching that. I didn't remember that you could
pass startup options like that. For that example,
process_startup_options() actually happens so late that we do not have
the intended value of log_connections until after we've already taken
all of the timings. As such, I propose we take those measurements
unconditionally (since we established they aren't too expensive when
compared to the cost of actually forking the backend) and then only
emit the message if log_connections is "timings". We are guaranteed to
have the intended value of log_connections by the time we are
ready-for-query. I've implemented this in [1]/messages/by-id/CAAKRu_aoerKAOYKkc7-JGq2bixrYTbViK_7EeLNhFUGoT_omFw@mail.gmail.com.
- Melanie
[1]: /messages/by-id/CAAKRu_aoerKAOYKkc7-JGq2bixrYTbViK_7EeLNhFUGoT_omFw@mail.gmail.com
On Tue, Mar 4, 2025 at 3:25 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
I don't
see any other TAP tests that are testing connections. There are a
whole bunch of auth tests, I suppose.
I was going to suggest src/test/authentication, yeah. Not perfect, but
better than nothing?
--Jacob
Hi,
On Tue, Mar 04, 2025 at 05:47:26PM -0500, Melanie Plageman wrote:
On Mon, Mar 3, 2025 at 9:07 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:I did not imagine that much ;-) I was just seeing this code being duplicated
and just thought about to avoid the duplication. But now that I read your comments
above then I think we could just macro-ize the child_type check (as you mentioned
up-thread). That would avoid the risk to forget to update the 3 locations doing the
exact same check should we add a new child type in the game.Is there a word we could use to describe what B_BACKEND and
B_WAL_SENDER have in common? They are the only backend types that will
go through the kind of external connection establishment steps (I
think), but I don't know a very accurate way to make that distinction
(which is required to come up with a useful macro name).
Yeah, what about IS_CONNECTION_TIMED_BACKEND?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Tue, Mar 04, 2025 at 06:25:42PM -0500, Melanie Plageman wrote:
Attached v9 implements log_connections as an enum with a new third
value "timings".On Mon, Mar 3, 2025 at 11:14 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:On Fri, Feb 28, 2025 at 05:52:35PM -0500, Melanie Plageman wrote:
We have to be really careful about when log_connections is actually
set before we use it -- I fixed some issues with that.Good catch! Yeah for the EXEC_BACKEND case we need to wait that read_nondefault_variables()
in SubPostmasterMain() is executed.Due to this and the bug Fujii-san reported with passing options to
psql (process_startup_options() is called too late for us to use the
value of log_connections as a gate for getting any of the timings),
I've decided to get the time regardless of the setting of
log_connections. This means we can be sure to have valid values at the
end when the message needs to be emitted.
I think that makes sense given what has been said in [1]/messages/by-id/bgn52vsoxoqato3l3wqsc3vqyd6xbkvd6ylhmh2yd5m4srjn3n@j5kocng6bfa2.
I was wondering if we should remove the backend type condition
(B_WAL_SENDER/B_BACKEND) too. It is only guarding capturing the fork
start and end time. That's just two measurements. Maybe we should just
always get fork start and end time. Spending extra time in forking due
to excess CPU activity (or some other reason) would be relevant for
all backend types, not just wal sender and client backends. We do only
want to log it in those cases, though...
Yeah, I'm on the fence for this one. OTOH that sounds strange to "collect"
data that won't be used for some backend types. So I'm tempted to vote to
keep it as it is.
+bool +check_log_connections(char **newval, void **extra, GucSource source) +{This function is pretty close to check_debug_io_direct() for example and its
overall content, memory management, looks good to me. I just have a few
following comments about it.Yep, one more reason I think PGC_SET is a good idea. Code deduplication.
+1
+typedef enum ConnectionLogOption +{ + LOG_CONNECTION_RECEIVED = (1 << 0), + LOG_CONNECTION_AUTHENTICATED = (1 << 1), + LOG_CONNECTION_AUTHORIZED = (1 << 2), + LOG_CONNECTION_DISCONNECTED = (1 << 3), +} ConnectionLogOption;I wonder if it would make sense to add LOG_CONNECTION_ALL here
(LOG_CONNECTION_RECEIVED | LOG_CONNECTION_AUTHENTICATED ..)That sounds a better place (than defining "all" in check_log_connections()) to
me. It's also how it is done in MonotonicFunction.The attached patch
Thanks for the new patch!
doesn't work exactly the same because it is a
regular enum and not a string list (so no "all")
Yeah, does not make sense anymore.
, but I do still have
a LogConnectionOption. The user interface is that timings includes all
the same messages as "on", however for the enum, I've defined it like
this:typedef enum LogConnectionOption
{
LOG_CONNECTION_BASIC = (1 << 0),
LOG_CONNECTION_TIMINGS = (1 << 1),
} LogConnectionOption;and then in the enum array, I have this
static const struct config_enum_entry log_connections_options[] = {
{"timings", LOG_CONNECTION_TIMINGS | LOG_CONNECTION_BASIC, false},
works for me.
Looking at the enum array:
"
static const struct config_enum_entry log_connections_options[] = {
{"off", 0, false},
{"on", LOG_CONNECTION_BASIC, false},
{"true", LOG_CONNECTION_BASIC, true},
{"false", 0, true},
{"yes", LOG_CONNECTION_BASIC, true},
{"no", 0, true},
{"timings", LOG_CONNECTION_TIMINGS | LOG_CONNECTION_BASIC, false},
{"1", LOG_CONNECTION_BASIC, true},
{"0", 0, true},
{NULL, 0, false}
};
"
and at parse_bool_with_len(), then it looks like it used to work with
log_connections set to things like "y, ye, yes, t, tr, tru, true, f, fa, fal,
fals, false, n and no" but now we want the full words.
I wonder if we should go so deep in the backward compatibility though. Maybe
that's worth to do if simple enough? (not sure how complicated it would be).
The case-insensitive is preserved, I mean "trUe" still works with the patch.
I think the way I've done it makes sense because the "business logic"
of how "timings" includes all the messages from "on" doesn't have to
pollute the code and doesn't take away our ability to use the enum
values independently.
Yeah, I like the way it's done.
When I want to emit a regular logging message, I can just check
if (log_connections & LOG_CONNECTION_BASIC)
which will work for "timings" too, but without taking away the
flexibility to use the enum values discretely.
yup.
Also not sure if it's worth adding a "MAX" (like it's done for relopt_kind)
because we'd probably not go over it anyay."
/* some compilers treat enums as signed ints, so we can't use 1 << 31 */
RELOPT_KIND_MAX = (1 << 30)That actually seems kind of broken -- what if it is on a platform
where integers are 16 bits?
Yeah, good point. Looks like a check on INT_MAX or such is missing. That's
probably worth another thread. I'll do that.
All the TAP tests have been modified that way:
-$node->append_conf('postgresql.conf', "log_connections = on"); +$node->append_conf('postgresql.conf', "log_connections = 'all'");Is it worh to add some checks for the other values?
Now we are back to only having 3 values,
but I was thinking if we
wanted to add some test exercising "timings". We can't test any
specific values of any of the durations emitted, so it would have to
just be a test that that log message was emitted.
yeah...
=== 1
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld",
+ (long int) INSTR_TIME_GET_MILLISEC(total_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration),
+ (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));
I wonder if doing:
+ errmsg("connection establishment times (ms): total: %.3f, fork: %.3f, authentication: %.3f",
+ (double) INSTR_TIME_GET_NANOSEC(total_duration) / 1000000.0,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.fork_duration) / 1000000.0,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.auth_duration) / 1000000.0));
wouln't be better. For example, on my machine I'd get:
connection establishment times (ms): total: 13.537, fork: 0.619, authentication: 0.267
instead of:
connection establishment times (ms): total: 13, fork: 0, authentication: 0
[1]: /messages/by-id/bgn52vsoxoqato3l3wqsc3vqyd6xbkvd6ylhmh2yd5m4srjn3n@j5kocng6bfa2
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Wed, Mar 5, 2025 at 5:36 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
Looking at the enum array:
"
static const struct config_enum_entry log_connections_options[] = {
{"off", 0, false},
{"on", LOG_CONNECTION_BASIC, false},
{"true", LOG_CONNECTION_BASIC, true},
{"false", 0, true},
{"yes", LOG_CONNECTION_BASIC, true},
{"no", 0, true},
{"timings", LOG_CONNECTION_TIMINGS | LOG_CONNECTION_BASIC, false},
{"1", LOG_CONNECTION_BASIC, true},
{"0", 0, true},
{NULL, 0, false}
};
"
and at parse_bool_with_len(), then it looks like it used to work with
log_connections set to things like "y, ye, yes, t, tr, tru, true, f, fa, fal,
fals, false, n and no" but now we want the full words.
Yes, this is a bit unfortunate, but I don't think we can accept t, f,
fa, etc. There is precedent for changing a boolean GUC to an enum --
synchronous_commit was like this.
I wonder if we should go so deep in the backward compatibility though. Maybe
that's worth to do if simple enough? (not sure how complicated it would be).
Yea, I don't think it makes sense to do more than the ones included in
that array.
+ ereport(LOG, + errmsg("connection establishment times (ms): total: %ld, fork: %ld, authentication: %ld", + (long int) INSTR_TIME_GET_MILLISEC(total_duration), + (long int) INSTR_TIME_GET_MILLISEC(conn_timing.fork_duration), + (long int) INSTR_TIME_GET_MILLISEC(conn_timing.auth_duration)));I wonder if doing:
+ errmsg("connection establishment times (ms): total: %.3f, fork: %.3f, authentication: %.3f", + (double) INSTR_TIME_GET_NANOSEC(total_duration) / 1000000.0, + (double) INSTR_TIME_GET_NANOSEC(conn_timing.fork_duration) / 1000000.0, + (double) INSTR_TIME_GET_NANOSEC(conn_timing.auth_duration) / 1000000.0));wouln't be better. For example, on my machine I'd get:
connection establishment times (ms): total: 13.537, fork: 0.619, authentication: 0.267
instead of:
connection establishment times (ms): total: 13, fork: 0, authentication: 0
Good idea. I've done this in my branch which I will post as patches later.
This morning, Andres reminded me off-list that we actually can still
support unquoted off and on values (as well as other single word
tokens like true, 1, etc) even if we transitioned to a string/list GUC
type. He also pointed out that tab-completion would be basically
useless for log_connections since it is not allowed to be set once the
connection is already established.
As such, I wonder if my PGC_SET idea is not worth the effort and I
should revise the earlier patch version which specified the stages but
allow for on, off, true, yes, 1 for backwards compatibility and not
include disconnections (so we don't remove the log_disconnections GUC
this release).
That would allow
log_connections = on
log_connections = off
as well as
log_connections = 'received, authenticated'
and even
log_connections = received
- Melanie
On Wed, Mar 5, 2025 at 10:46 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:
As such, I wonder if my PGC_SET idea is not worth the effort and I
should revise the earlier patch version which specified the stages but
allow for on, off, true, yes, 1 for backwards compatibility and not
include disconnections (so we don't remove the log_disconnections GUC
this release).
Okay! Attached v10 does this. It turns log_connections into a string
but supports all the common values of the current log_connections
boolean.
So, you can specify
log_connections = authorized,authenticated
log_connections = 'all'
log_connections = on
etc
And only 'all' has to be quoted when not specified as a startup option
(which is true for other enum gucs that have 'all' as an option
because ALL is a Postgres keyword).
I've left log_disconnections alone.
I still need to comprehensively manually test it myself (so there
might still be bugs) and write a bit of test coverage for the new
values. The docs can probably also use some more massaging.
But, this is the first version of the patch where I am happy with the
interface and with the code. It isn't a breaking change (since on,
off, true, false, etc still work), so I think it is still okay to
target 18.
I went back through the thread and tried to be sure I addressed all of
the outstanding review comments on previous versions (since I
resurrected some previously dead patch features), but I can't be sure
I caught it all. Please don't hesitate to call out anything I missed.
I didn't enumerate all the possibilities in postgresql.conf.sample as
Fujii-san recommended in a previous review, because I didn't see many
other enum GUCs doing that. I am open to changing that if folks feel
strongly, though.
In 0002, because we take the timings for all wal sender and client
connection backends, now the new log message emitted in 0002 is more
like a stage, so I've named that option "ready_for_query". As such, I
wonder if I should change the output message to reflect that. What do
you think?
- Melanie
Attachments:
v10-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v10-0002-Add-connection-establishment-duration-logging.patchDownload
From 81fdc8932f5caff49cb89461be010bb84c20c234 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:05:19 -0500
Subject: [PATCH v10 2/2] Add connection establishment duration logging
Add log_connections option 'ready_for_query' which logs durations of
several key parts of connection establishment and setup the first time
the backend is ready for query.
For an incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. This logging provides visibility
into authentication and fork duration as well as the end-to-end
connection establishment and initialization time.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData instead of simply saved in backend local memory.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 9 +++++
src/backend/postmaster/launch_backend.c | 39 +++++++++++++++++++++
src/backend/postmaster/postmaster.c | 6 ++++
src/backend/tcop/backend_startup.c | 1 +
src/backend/tcop/postgres.c | 22 ++++++++++++
src/backend/utils/init/globals.c | 3 ++
src/backend/utils/init/postinit.c | 8 +++++
src/include/miscadmin.h | 9 +++++
src/include/portability/instr_time.h | 9 +++++
src/include/tcop/backend_startup.h | 45 +++++++++++++++++++++++--
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 149 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a15ff198861..512408633b9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7362,6 +7362,15 @@ local0.* /var/log/postgresql
</entry>
</row>
+ <row>
+ <entry><literal>ready_for_query</literal></entry>
+ <entry>
+ Logs the duration of connection establishment and setup as well as
+ several establishment component durations. At this point, the
+ backend is ready for query.
+ </entry>
+ </row>
+
<row>
<entry><literal>all</literal></entry>
<entry>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..e5587354c02 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (IsConnectionBackend(child_type))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_started);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,21 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /*
+ * Transfer over any timing data captured by postmaster that may be
+ * needed for log_connections.
+ */
+ if (IsConnectionBackend(child_type))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ /* Calculate fork duration since we have start and end times. */
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(data->fork_started);
+
+ /* Will be used to calculate total connection setup duration. */
+ conn_timing.creation_time = data->socket_created;
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,6 +605,7 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ instr_time fork_ended;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
@@ -615,6 +635,9 @@ SubPostmasterMain(int argc, char *argv[])
if (!found)
elog(ERROR, "unknown child kind %s", child_kind);
+ if (IsConnectionBackend(child_type))
+ INSTR_TIME_SET_CURRENT(fork_ended);
+
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
@@ -648,6 +671,22 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /*
+ * Transfer over any timing data captured by postmaster that may be needed
+ * for log_connections.
+ */
+ if (IsConnectionBackend(child_type))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ /* Calculate fork duration since we have start and end times. */
+ conn_timing.fork_duration = fork_ended;
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration, data->fork_started);
+
+ /* Will be used to calculate total connection setup duration. */
+ conn_timing.creation_time = data->socket_created;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 616cd8be771..c12e8b7a252 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -3477,6 +3477,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment and setup total duration).
+ */
+ INSTR_TIME_SET_CURRENT(startup_data.socket_created);
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index 8a4f88d2240..436408d9093 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -993,6 +993,7 @@ validate_log_connections_options(List *elemlist)
{"received", LOG_CONNECTION_RECEIVED},
{"authenticated", LOG_CONNECTION_AUTHENTICATED},
{"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"ready_for_query", LOG_CONNECTION_READY_FOR_QUERY},
{"all", LOG_CONNECTION_ALL},
};
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..0a529136244 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -66,6 +66,7 @@
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinval.h"
+#include "tcop/backend_startup.h"
#include "tcop/fastpath.h"
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
@@ -4153,6 +4154,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4609,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ log_connections & LOG_CONNECTION_READY_FOR_QUERY &&
+ IsConnectionBackend(MyBackendType))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(conn_timing.creation_time);
+
+ ereport(LOG,
+ errmsg("connection establishment times (ms): total: %.3f, fork: %.3f, authentication: %.3f",
+ (double) INSTR_TIME_GET_NANOSEC(total_duration) / NS_PER_MS,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.fork_duration) / NS_PER_MS,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.auth_duration) / NS_PER_MS));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..9c59f31fc84 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -24,6 +24,7 @@
#include "miscadmin.h"
#include "postmaster/postmaster.h"
#include "storage/procnumber.h"
+#include "tcop/backend_startup.h"
ProtocolVersion FrontendProtocol;
@@ -43,6 +44,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index dc6484bb092..7c1ab501395 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -192,9 +192,14 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -253,6 +258,9 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..ec5ed2235e6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
@@ -394,6 +396,13 @@ extern PGDLLIMPORT BackendType MyBackendType;
(AmAutoVacuumLauncherProcess() || \
AmLogicalSlotSyncWorkerProcess())
+/*
+ * Backend types that are spawned by the postmaster to serve a client
+ * connection.
+ */
+#define IsConnectionBackend(backend_type) \
+ (backend_type == B_BACKEND || backend_type == B_WAL_SENDER)
+
extern const char *GetBackendTypeDesc(BackendType backendType);
extern void SetDatabasePath(const char *path);
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 5659229f45e..786daa89d02 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
extern PGDLLIMPORT int log_connections;
@@ -39,13 +41,25 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the postmaster initiates a fork of a backend process.
+ * Only used for client and wal sender connections.
+ */
+ instr_time fork_started;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ instr_time socket_created;
} BackendStartupData;
/*
* Granular control over which messages to log for the log_connections GUC.
*
- * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
- * establishment and setup where we may emit a log message.
+ * RECEIVED, AUTHENTICATED, AUTHORIZED, and READY_FOR_QUERY are different
+ * stages of connection establishment and setup where we may emit a log message.
*
* ALL is a convenience alias for when all stages should be logged.
*
@@ -57,14 +71,39 @@ typedef enum LogConnectionOption
LOG_CONNECTION_RECEIVED = (1 << 0),
LOG_CONNECTION_AUTHENTICATED = (1 << 1),
LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_READY_FOR_QUERY = (1 << 3),
LOG_CONNECTION_ON =
LOG_CONNECTION_RECEIVED |
LOG_CONNECTION_AUTHENTICATED |
LOG_CONNECTION_AUTHORIZED,
LOG_CONNECTION_ALL =
- LOG_CONNECTION_ON,
+ LOG_CONNECTION_ON |
+ LOG_CONNECTION_READY_FOR_QUERY,
} LogConnectionOption;
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders.
+ *
+ * Used to emit log messages according to the options specified in the
+ * log_connections GUC.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created. This is used to log the
+ * total connection establishment and setup time from accept() to first
+ * being ready for query.
+ */
+ instr_time creation_time;
+
+ /* How long it took the backend to be forked. */
+ instr_time fork_duration;
+
+ /* How long authentication took */
+ instr_time auth_duration;
+} ConnectionTiming;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 29e52b64994..3da74999634 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
v10-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v10-0001-Modularize-log_connections-output.patchDownload
From 68a1bf197a5f39e19a1f3de3c4052ced3cee1126 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:02:45 -0500
Subject: [PATCH v10 1/2] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC of the
connection stages to log.
The current log_connections options are 'received', 'authenticated', and
'authorized'. The empty string disables all connection logging. 'all'
enables all available connection logging.
This gives users more control over the volume of connection logging.
For backwards compatability, the most common values for the
log_connections boolean are still supported ('on', 'off', 1, 0, true,
false).
This patch has much of the same behavior as [1] but was developed
independently.
[1] https://postgr.es/m/flat/CAA8Fd-qCB96uwfgMKrzfNs32mqqysi53yZFNVaRNJ6xDthZEgA%40mail.gmail.com
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 71 ++++++++-
src/backend/libpq/auth.c | 9 +-
src/backend/postmaster/postmaster.c | 1 -
src/backend/tcop/backend_startup.c | 145 +++++++++++++++++-
src/backend/utils/init/postinit.c | 3 +-
src/backend/utils/misc/guc_tables.c | 21 +--
src/backend/utils/misc/postgresql.conf.sample | 4 +-
src/include/postmaster/postmaster.h | 1 -
src/include/tcop/backend_startup.h | 26 ++++
src/include/utils/guc_hooks.h | 2 +
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 260 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..a15ff198861 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,81 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes the specified stages of each connection attempt to the server to
+ be logged. The default is the empty string, <literal>''</literal>,
+ which disables all connection logging. The following are the options
+ that may be specified:
+ </para>
+
+ <table id="log-connections-stages">
+ <title>Log Connection Stages</title>
+ <tgroup cols="2">
+ <colspec colname="col1" colwidth="1*"/>
+ <colspec colname="col2" colwidth="2*"/>
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>received</literal></entry>
+ <entry>Logs receipt of a connection.</entry>
+ </row>
+
+ <row>
+ <entry><literal>authenticated</literal></entry>
+ <entry>
+ Logs the original identity used by an authentication method to
+ identify a user. Failed authentication is always logged regardless
+ of the value of this setting.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>authorized</literal></entry>
+ <entry>
+ Logs successful completion of authorization. At this point the
+ connection has been fully established.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>all</literal></entry>
+ <entry>
+ A convenience alias. If specified, <literal>all</literal> must be
+ quoted because it is a <productname>PostgreSQL</productname>
+ keyword. If <literal>all</literal> is specified in a list of other
+ options, all connection logging will be enabled.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ For the purposes of backwards compatability, <literal>on</literal>,
+ <literal>off</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>yes</literal>,
+ <literal>no</literal>, <literal>1</literal>, and <literal>0</literal>
+ are still supported. The positive values emit logs for the
+ <literal>received</literal>, <literal>authenticated</literal>, and
+ <literal>authorized</literal> stages.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..ea0ea48805b 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -38,6 +38,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/ipc.h"
+#include "tcop/backend_startup.h"
#include "utils/memutils.h"
/*----------------------------------------------------------------
@@ -317,7 +318,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authenticated' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +351,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +635,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 5dd3b6a4fd4..616cd8be771 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,6 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
bool enable_bonjour = false;
char *bonjour_name;
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..8a4f88d2240 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -34,13 +34,17 @@
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
+#include "utils/varlena.h"
/* GUCs */
bool Trace_connection_negotiation = false;
+int log_connections = 0;
+char *log_connections_string = NULL;
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
@@ -48,6 +52,7 @@ static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void process_startup_packet_die(SIGNAL_ARGS);
static void StartupPacketTimeoutHandler(void);
+static int validate_log_connections_options(List *elemlist);
/*
* Entry point for a new backend process.
@@ -201,8 +206,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIVED)
{
if (remote_port[0])
ereport(LOG,
@@ -924,3 +929,139 @@ StartupPacketTimeoutHandler(void)
{
_exit(1);
}
+
+/*
+ * Helper for the log_connections GUC check hook.
+ *
+ * `elemlist` is the listified string input passed to check_log_connections().
+ * check_log_connections() is responsible for cleaning up `elemlist`.
+ *
+ * validate_log_connections_options() returns the flags that should be stored
+ * in the log_connections GUC by the assign_hook or -1 if the input is not
+ * valid and we should error out.
+ */
+static int
+validate_log_connections_options(List *elemlist)
+{
+ int flags = 0;
+ ListCell *l;
+ char *item;
+
+ /* For backwards compatability, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
+
+ /* If an empty string was passed, we're done. */
+ if (list_length(elemlist) == 0)
+ return 0;
+
+ /*
+ * Now check for the backwards compatability options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatability option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
+ GUC_check_errdetail("Cannot specify log_connections option \"%s\" in a list with other options.", item);
+ return -1;
+ }
+
+ return option.val;
+ }
+
+ /* Now check the stage-specific options. */
+ foreach(l, elemlist)
+ {
+ static const struct config_enum_entry options[] = {
+ {"received", LOG_CONNECTION_RECEIVED},
+ {"authenticated", LOG_CONNECTION_AUTHENTICATED},
+ {"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"all", LOG_CONNECTION_ALL},
+ };
+
+ item = lfirst(l);
+ for (size_t i = 0; i < lengthof(options); i++)
+ {
+ struct config_enum_entry option = options[i];
+
+ if (pg_strcasecmp(item, option.name) == 0)
+ {
+ flags |= option.val;
+ goto next;
+ }
+ }
+
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ return -1;
+
+next: ;
+ }
+
+ return flags;
+}
+
+
+/*
+ * GUC check hook for log_connections
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ int flags;
+ char *rawstring;
+ List *elemlist;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitIdentifierString(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Validation logic is all in the helper */
+ flags = validate_log_connections_options(elemlist);
+
+ /* Time for cleanup and allocating `extra` if we succeeded */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ /* We didn't succeed */
+ if (flags == -1)
+ return false;
+
+ /* Save the flags in *extra, for use by assign_log_connections */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+}
+
+/*
+ * GUC assign hook for log_connections
+ */
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..dc6484bb092 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -54,6 +54,7 @@
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -252,7 +253,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..71070465379 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4886,6 +4877,18 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about events during connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..aca610f4fad 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections=all". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,7 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = ''
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..aa786bfacf3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,6 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..5659229f45e 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -16,6 +16,8 @@
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
+extern PGDLLIMPORT int log_connections;
+extern PGDLLIMPORT char *log_connections_string;
/*
* CAC_state is passed from postmaster to the backend process, to indicate
@@ -39,6 +41,30 @@ typedef struct BackendStartupData
CAC_state canAcceptConnections;
} BackendStartupData;
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ *
+ * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
+ * establishment and setup where we may emit a log message.
+ *
+ * ALL is a convenience alias for when all stages should be logged.
+ *
+ * ON is backwards compatability alias for the connection events that were
+ * logged in Postgres versions < 18.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIVED |
+ LOG_CONNECTION_AUTHENTICATED |
+ LOG_CONNECTION_AUTHORIZED,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
+} LogConnectionOption;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..29e52b64994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1555,6 +1555,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
Hi,
On Wed, Mar 05, 2025 at 06:40:10PM -0500, Melanie Plageman wrote:
On Wed, Mar 5, 2025 at 10:46 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:As such, I wonder if my PGC_SET idea is not worth the effort and I
should revise the earlier patch version which specified the stages but
allow for on, off, true, yes, 1 for backwards compatibility and not
include disconnections (so we don't remove the log_disconnections GUC
this release).Okay! Attached v10 does this.
Great, thanks for it!
It turns log_connections into a string
but supports all the common values of the current log_connections
boolean.So, you can specify
log_connections = authorized,authenticated
Yeah, when used like psql -d "options=-clog_connections=authorized,authenticated"
but needs to be log_connections = 'authorized,authenticated' in the postgresql.conf
file. That makes fully sense.
But, this is the first version of the patch where I am happy with the
interface and with the code. It isn't a breaking change (since on,
off, true, false, etc still work), so I think it is still okay to
target 18.
A few comments about 0001:
=== 1
we have this:
+ /* For backwards compatability, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
and
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIVED |
+ LOG_CONNECTION_AUTHENTICATED |
+ LOG_CONNECTION_AUTHORIZED,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
+} LogConnectionOption;
what do you think about also doing?
static const struct config_enum_entry compat_options[] = {
- {"off", 0},
- {"false", 0},
- {"no", 0},
- {"0", 0},
+ {"off", LOG_CONNECTION_OFF},
+ {"false", LOG_CONNECTION_OFF},
+ {"no", LOG_CONNECTION_OFF},
+ {"0", LOG_CONNECTION_OFF},
and
typedef enum LogConnectionOption
{
+ LOG_CONNECTION_OFF = 0,
LOG_CONNECTION_RECEIVED = (1 << 0),
and
/* If an empty string was passed, we're done. */
if (list_length(elemlist) == 0)
return LOG_CONNECTION_OFF;
=== 2
- if (Log_connections && status == STATUS_OK &&
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED &&
+ status == STATUS_OK &&
Worth to add extra parentheses around (log_connections & LOG_CONNECTION_AUTHENTICATED)?
Not necessary as bitwise AND has higher precedence than logical AND but I think
it makes the code more readable.
=== 3
+ /* If an empty string was passed, we're done. */
+ if (list_length(elemlist) == 0)
+ return 0;
+
+ /*
+ * Now check for the backwards compatability options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatability option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
what about storing the list_length(elemlist) at the start to avoid the multiple
list_length(elemlist) calls?
=== 4
+ /* If an empty string was passed, we're done. */
s/done./done/? (Looking at the one line comments around)
=== 5
+ /* Now check the stage-specific options. */
s/options./options/? (Looking at the one line comments around)
=== 6
+ /* For backwards compatability, we accept these tokens by themselves */
Typo: compatability/compatibility
, seen multiple times:
$ git grep compatability
doc/src/sgml/config.sgml: For the purposes of backwards compatability, <literal>on</literal>,
src/backend/tcop/backend_startup.c: /* For backwards compatability, we accept these tokens by themselves */
src/backend/tcop/backend_startup.c: * Now check for the backwards compatability options. They must always be
src/backend/tcop/backend_startup.c: * backwards compatability option and other options are also specified.
src/include/tcop/backend_startup.h: * ON is backwards compatability alias for the connection events that were
=== 7
+ /* Time for cleanup and allocating `extra` if we succeeded */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ /* We didn't succeed */
+ if (flags == -1)
+ return false;
I feel the first comment is confusing, maybe the "allocating `extra` if we succeeded"
should be put after the "if (flags == -1)" part?
The logic in 0001 looks good to me, also I did a few tests and that looks ok
seen from here.
In 0002, because we take the timings for all wal sender and client
connection backends, now the new log message emitted in 0002 is more
like a stage, so I've named that option "ready_for_query". As such, I
wonder if I should change the output message to reflect that. What do
you think?
hmm, I'm tempted to vote for "timings". I understand your point but "timings"
more directly communicates that this option enables timing measurements,
whereas "ready_for_query" describes just one particular state.
I think I could vote for "ready_for_query" should the GUC be log_connection_stages
(and not log_connections).
I also think that we can keep "reported_first_ready_for_query", "send_ready_for_query"
in the code though (even if the GUC option is changed to "timings").
A few comments about 0002:
==== 8
+ if (IsConnectionBackend(child_type))
IsConnectionBackend is fine by me.
=== 9
+ /* Calculate fork duration since we have start and end times. */
s/times./times/? (Looking at the one line comments around)
=== 10
+ /* Will be used to calculate total connection setup duration. */
s/duration./duration/? (Looking at the one line comments around)
=== 11
+ /* How long it took the backend to be forked. */
s/forked./forked/? (Looking at the one line comments around)
=== 12
+ if (!reported_first_ready_for_query &&
+ log_connections & LOG_CONNECTION_READY_FOR_QUERY &&
+ IsConnectionBackend(MyBackendType))
same as comment === 2 above.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Thanks for the continued review!
Attached v11 has a test added (passes locally but fails in CI, so I
have to fix that).
I still need to do some more manual testing and validation.
On Thu, Mar 6, 2025 at 9:56 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
<-- snip -->
what do you think about also doing?
static const struct config_enum_entry compat_options[] = { - {"off", 0}, - {"false", 0}, - {"no", 0}, - {"0", 0}, + {"off", LOG_CONNECTION_OFF}, + {"false", LOG_CONNECTION_OFF}, + {"no", LOG_CONNECTION_OFF}, + {"0", LOG_CONNECTION_OFF},and
typedef enum LogConnectionOption
{
+ LOG_CONNECTION_OFF = 0,
LOG_CONNECTION_RECEIVED = (1 << 0),
and/* If an empty string was passed, we're done. */
if (list_length(elemlist) == 0)
return LOG_CONNECTION_OFF;
Yes, I do notice that most GUCs with enums that have an "off" switch
also have a "NONE" or "OFF" enumeration value. This is to contrast it
with a GUC that doesn't have an "off" switch (like wal_level). So, I
think you're right that we should have something to indicate you can
turn off log_connections.
However, I was a bit torn about LOG_CONNECTION_OFF vs
LOG_CONNECTION_NONE. We have LOG_CONNECTION_ON because it is a
legitimately distinct alias for the log messages emitted by < pg 18
when log_connections == on. However, LOG_CONNECTION_NONE and
LOG_CONNECTION_OFF would be the same. As such, I prefer
LOG_CONNECTION_NONE because it makes more sense in the context of
where the enumeration is defined. I've added this and used it in the
compat_options array. What do you think? Is it too confusing?
- if (Log_connections && status == STATUS_OK && + if (log_connections & LOG_CONNECTION_AUTHENTICATED && + status == STATUS_OK &&Worth to add extra parentheses around (log_connections & LOG_CONNECTION_AUTHENTICATED)?
Not necessary as bitwise AND has higher precedence than logical AND but I think
it makes the code more readable.
Done in all places (I think).
+ /* If an empty string was passed, we're done. */ + if (list_length(elemlist) == 0) + return 0; +
< -- snip -->
what about storing the list_length(elemlist) at the start to avoid the multiple
list_length(elemlist) calls?
Since it would only be called twice and it has the length stored in
the List struct, I prefer it as is -- without an extra local variable.
+ /* If an empty string was passed, we're done. */
s/done./done/? (Looking at the one line comments around)
I've addressed all of these comment punctuations you've mentioned. Thanks!
=== 6
+ /* For backwards compatability, we accept these tokens by themselves */
Typo: compatability/compatibility
Whoops! Thanks. I forgot to spell check my commit message. Done that now too.
How do you find spelling mistakes in patches usually? I tried `git
show | aspell list` -- but of course that includes lots of variables
and such that aren't actually spelling mistakes.
+ /* Time for cleanup and allocating `extra` if we succeeded */ + pfree(rawstring); + list_free(elemlist); + + /* We didn't succeed */ + if (flags == -1) + return false;I feel the first comment is confusing, maybe the "allocating `extra` if we succeeded"
should be put after the "if (flags == -1)" part?
Fixed.
In 0002, because we take the timings for all wal sender and client
connection backends, now the new log message emitted in 0002 is more
like a stage, so I've named that option "ready_for_query". As such, I
wonder if I should change the output message to reflect that. What do
you think?hmm, I'm tempted to vote for "timings". I understand your point but "timings"
more directly communicates that this option enables timing measurements,
whereas "ready_for_query" describes just one particular state.
Well, I actually didn't want to call it "timings" because it implies
that we only measure the timings when it is specified. But we actually
get the timings unconditionally for client backends and wal sender.
Don't you think it is more confusing for the user if we call it
timings and users think if they don't include that timings aren't
measured?
I didn't change it in the attached version 11. I wanted to discuss
more before making that decision.
Also, I thought changing the message output to say "ready for query"
somewhere in it makes it more clear when that message is actually
emitted in a connection's lifetime. What do you think?
I think I could vote for "ready_for_query" should the GUC be log_connection_stages
(and not log_connections).
All the existing connection messages are at certain stages and fall in
this category. However, were we to add more that don't fall in that
category, we'd have to think about how to modularize them. I'm not
sure if we could mix connection stages and other assorted message
types easily without the groupings you've mentioned in the past. What
we don't want to do is make a change that makes that harder to do in
the future.
- Melanie
Attachments:
v11-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v11-0002-Add-connection-establishment-duration-logging.patchDownload
From f0040ff957176a3dec61cacc7771945e6e2ccf8d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:05:19 -0500
Subject: [PATCH v11 2/3] Add connection establishment duration logging
Add log_connections option 'ready_for_query' which logs durations of
several key parts of connection establishment and setup the first time
the backend is ready for query.
For an incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. This logging provides visibility
into authentication and fork duration as well as the end-to-end
connection establishment and initialization time.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData instead of simply saved in backend local memory.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 9 ++++
src/backend/postmaster/launch_backend.c | 39 ++++++++++++++++
src/backend/postmaster/postmaster.c | 6 +++
src/backend/tcop/backend_startup.c | 1 +
src/backend/tcop/postgres.c | 22 +++++++++
src/backend/utils/init/globals.c | 3 ++
src/backend/utils/init/postinit.c | 8 ++++
src/include/miscadmin.h | 9 ++++
src/include/portability/instr_time.h | 9 ++++
src/include/tcop/backend_startup.h | 45 +++++++++++++++++--
.../postmaster/t/002_connection_limits.pl | 8 +++-
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b97707a7ec2..f424b3f215b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7362,6 +7362,15 @@ local0.* /var/log/postgresql
</entry>
</row>
+ <row>
+ <entry><literal>ready_for_query</literal></entry>
+ <entry>
+ Logs the duration of connection establishment and setup as well as
+ several establishment component durations. At this point, the
+ backend is ready for query.
+ </entry>
+ </row>
+
<row>
<entry><literal>all</literal></entry>
<entry>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..097605b4070 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates fork for logging */
+ if (IsConnectionBackend(child_type))
+ INSTR_TIME_SET_CURRENT(((BackendStartupData *) startup_data)->fork_started);
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,21 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /*
+ * Transfer over any timing data captured by postmaster that may be
+ * needed for log_connections.
+ */
+ if (IsConnectionBackend(child_type))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ /* Calculate fork duration since we have start and end times */
+ conn_timing.fork_duration = INSTR_TIME_GET_DURATION_SINCE(data->fork_started);
+
+ /* Will be used to calculate total connection setup duration */
+ conn_timing.creation_time = data->socket_created;
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,6 +605,7 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ instr_time fork_ended;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
@@ -615,6 +635,9 @@ SubPostmasterMain(int argc, char *argv[])
if (!found)
elog(ERROR, "unknown child kind %s", child_kind);
+ if (IsConnectionBackend(child_type))
+ INSTR_TIME_SET_CURRENT(fork_ended);
+
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
@@ -648,6 +671,22 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /*
+ * Transfer over any timing data captured by postmaster that may be needed
+ * for log_connections.
+ */
+ if (IsConnectionBackend(child_type))
+ {
+ BackendStartupData *data = (BackendStartupData *) startup_data;
+
+ /* Calculate fork duration since we have start and end times */
+ conn_timing.fork_duration = fork_ended;
+ INSTR_TIME_SUBTRACT(conn_timing.fork_duration, data->fork_started);
+
+ /* Will be used to calculate total connection setup duration */
+ conn_timing.creation_time = data->socket_created;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b4f34c57a1b..d46ffa34151 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -3477,6 +3477,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment and setup total duration).
+ */
+ INSTR_TIME_SET_CURRENT(startup_data.socket_created);
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index 84db47231f6..399562bb599 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -997,6 +997,7 @@ validate_log_connections_options(List *elemlist)
{"received", LOG_CONNECTION_RECEIVED},
{"authenticated", LOG_CONNECTION_AUTHENTICATED},
{"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"ready_for_query", LOG_CONNECTION_READY_FOR_QUERY},
{"all", LOG_CONNECTION_ALL},
};
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..4e74f309c47 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -66,6 +66,7 @@
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinval.h"
+#include "tcop/backend_startup.h"
#include "tcop/fastpath.h"
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
@@ -4153,6 +4154,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4609,26 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_READY_FOR_QUERY) &&
+ IsConnectionBackend(MyBackendType))
+ {
+ instr_time total_duration =
+ INSTR_TIME_GET_DURATION_SINCE(conn_timing.creation_time);
+
+ ereport(LOG,
+ errmsg("connection ready for query. durations (ms): total: %.3f, fork: %.3f, authentication: %.3f",
+ (double) INSTR_TIME_GET_NANOSEC(total_duration) / NS_PER_MS,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.fork_duration) / NS_PER_MS,
+ (double) INSTR_TIME_GET_NANOSEC(conn_timing.auth_duration) / NS_PER_MS));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index b844f9fdaef..9c59f31fc84 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -24,6 +24,7 @@
#include "miscadmin.h"
#include "postmaster/postmaster.h"
#include "storage/procnumber.h"
+#include "tcop/backend_startup.h"
ProtocolVersion FrontendProtocol;
@@ -43,6 +44,8 @@ volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
+ConnectionTiming conn_timing = {0};
+
int MyProcPid;
pg_time_t MyStartTime;
TimestampTz MyStartTimestamp;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index dc6484bb092..7c1ab501395 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -192,9 +192,14 @@ GetDatabaseTupleByOid(Oid dboid)
static void
PerformAuthentication(Port *port)
{
+ instr_time auth_start_time;
+
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ INSTR_TIME_SET_CURRENT(auth_start_time);
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -253,6 +258,9 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Calculate authentication duration for logging */
+ conn_timing.auth_duration = INSTR_TIME_GET_DURATION_SINCE(auth_start_time);
+
if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..ec5ed2235e6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -178,6 +178,8 @@ extern PGDLLIMPORT int MaxConnections;
extern PGDLLIMPORT int max_worker_processes;
extern PGDLLIMPORT int max_parallel_workers;
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
extern PGDLLIMPORT int commit_timestamp_buffers;
extern PGDLLIMPORT int multixact_member_buffers;
extern PGDLLIMPORT int multixact_offset_buffers;
@@ -394,6 +396,13 @@ extern PGDLLIMPORT BackendType MyBackendType;
(AmAutoVacuumLauncherProcess() || \
AmLogicalSlotSyncWorkerProcess())
+/*
+ * Backend types that are spawned by the postmaster to serve a client
+ * connection.
+ */
+#define IsConnectionBackend(backend_type) \
+ (backend_type == B_BACKEND || backend_type == B_WAL_SENDER)
+
extern const char *GetBackendTypeDesc(BackendType backendType);
extern void SetDatabasePath(const char *path);
diff --git a/src/include/portability/instr_time.h b/src/include/portability/instr_time.h
index f71a851b18d..48d7ff1bfad 100644
--- a/src/include/portability/instr_time.h
+++ b/src/include/portability/instr_time.h
@@ -184,6 +184,15 @@ GetTimerFrequency(void)
#define INSTR_TIME_ACCUM_DIFF(x,y,z) \
((x).ticks += (y).ticks - (z).ticks)
+static inline instr_time
+INSTR_TIME_GET_DURATION_SINCE(instr_time start_time)
+{
+ instr_time now;
+
+ INSTR_TIME_SET_CURRENT(now);
+ INSTR_TIME_SUBTRACT(now, start_time);
+ return now;
+}
#define INSTR_TIME_GET_DOUBLE(t) \
((double) INSTR_TIME_GET_NANOSEC(t) / NS_PER_S)
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 66d370c1e00..d4ad9f69086 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,6 +14,8 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "portability/instr_time.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
extern PGDLLIMPORT int log_connections;
@@ -39,13 +41,25 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the postmaster initiates a fork of a backend process.
+ * Only used for client and wal sender connections.
+ */
+ instr_time fork_started;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ instr_time socket_created;
} BackendStartupData;
/*
* Granular control over which messages to log for the log_connections GUC.
*
- * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
- * establishment and setup where we may emit a log message.
+ * RECEIVED, AUTHENTICATED, AUTHORIZED, and READY_FOR_QUERY are different
+ * stages of connection establishment and setup where we may emit a log message.
*
* ALL is a convenience alias for when all stages should be logged.
*
@@ -60,14 +74,39 @@ typedef enum LogConnectionOption
LOG_CONNECTION_RECEIVED = (1 << 0),
LOG_CONNECTION_AUTHENTICATED = (1 << 1),
LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_READY_FOR_QUERY = (1 << 3),
LOG_CONNECTION_ON =
LOG_CONNECTION_RECEIVED |
LOG_CONNECTION_AUTHENTICATED |
LOG_CONNECTION_AUTHORIZED,
LOG_CONNECTION_ALL =
- LOG_CONNECTION_ON,
+ LOG_CONNECTION_ON |
+ LOG_CONNECTION_READY_FOR_QUERY,
} LogConnectionOption;
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders.
+ *
+ * Used to emit log messages according to the options specified in the
+ * log_connections GUC.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created. This is used to log the
+ * total connection establishment and setup time from accept() to first
+ * being ready for query.
+ */
+ instr_time creation_time;
+
+ /* How long it took the backend to be forked */
+ instr_time fork_duration;
+
+ /* How long authentication took */
+ instr_time auth_duration;
+} ConnectionTiming;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/test/postmaster/t/002_connection_limits.pl b/src/test/postmaster/t/002_connection_limits.pl
index f19d6e73ac9..de35d85e23e 100644
--- a/src/test/postmaster/t/002_connection_limits.pl
+++ b/src/test/postmaster/t/002_connection_limits.pl
@@ -29,11 +29,12 @@ $node->connect_ok("",
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized/,
+ qr/connection ready for query/,
],);
$node->safe_psql(
'postgres',
- q[ALTER SYSTEM SET log_connections = received,authorized;
+ q[ALTER SYSTEM SET log_connections = received,authorized,ready_for_query;
SELECT pg_reload_conf();]);
$node->connect_ok("",
@@ -41,6 +42,7 @@ $node->connect_ok("",
log_like => [
qr/connection received/,
qr/connection authorized/,
+ qr/connection ready for query/,
],
log_unlike => [
qr/connection authenticated/,
@@ -56,6 +58,9 @@ $node->connect_ok("",
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized/,
+ ],
+ log_unlike => [
+ qr/connection ready for query/,
],);
$node->safe_psql(
@@ -68,6 +73,7 @@ $node->connect_ok("",
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized/,
+ qr/connection ready for query/,
],);
$node->safe_psql(
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 29e52b64994..3da74999634 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
v11-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v11-0001-Modularize-log_connections-output.patchDownload
From 2fb563dc2f95ff50bf492340cd1e1d10aa45eb94 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:02:45 -0500
Subject: [PATCH v11 1/3] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC of the
connection stages to log.
The current log_connections options are 'received', 'authenticated', and
'authorized'. The empty string disables all connection logging. 'all'
enables all available connection logging.
This gives users more control over the volume of connection logging.
For backwards compatibility, the most common values for the
log_connections boolean are still supported ('on', 'off', 1, 0, true,
false).
This patch has much of the same behavior as [1] but was developed
independently.
[1] https://postgr.es/m/flat/CAA8Fd-qCB96uwfgMKrzfNs32mqqysi53yZFNVaRNJ6xDthZEgA%40mail.gmail.com
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 71 +++++++-
src/backend/libpq/auth.c | 9 +-
src/backend/postmaster/postmaster.c | 1 -
src/backend/tcop/backend_startup.c | 152 +++++++++++++++++-
src/backend/utils/init/postinit.c | 3 +-
src/backend/utils/misc/guc_tables.c | 21 +--
src/backend/utils/misc/postgresql.conf.sample | 4 +-
src/include/postmaster/postmaster.h | 1 -
src/include/tcop/backend_startup.h | 29 ++++
src/include/utils/guc_hooks.h | 2 +
.../postmaster/t/002_connection_limits.pl | 50 +++++-
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 319 insertions(+), 25 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..b97707a7ec2 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,81 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes the specified stages of each connection attempt to the server to
+ be logged. The default is the empty string, <literal>''</literal>,
+ which disables all connection logging. The following are the options
+ that may be specified:
+ </para>
+
+ <table id="log-connections-stages">
+ <title>Log Connection Stages</title>
+ <tgroup cols="2">
+ <colspec colname="col1" colwidth="1*"/>
+ <colspec colname="col2" colwidth="2*"/>
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>received</literal></entry>
+ <entry>Logs receipt of a connection.</entry>
+ </row>
+
+ <row>
+ <entry><literal>authenticated</literal></entry>
+ <entry>
+ Logs the original identity used by an authentication method to
+ identify a user. Failed authentication is always logged regardless
+ of the value of this setting.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>authorized</literal></entry>
+ <entry>
+ Logs successful completion of authorization. At this point the
+ connection has been fully established.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>all</literal></entry>
+ <entry>
+ A convenience alias. If specified, <literal>all</literal> must be
+ quoted because it is a <productname>PostgreSQL</productname>
+ keyword. If <literal>all</literal> is specified in a list of other
+ options, all connection logging will be enabled.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ For the purposes of backwards compatibility, <literal>on</literal>,
+ <literal>off</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>yes</literal>,
+ <literal>no</literal>, <literal>1</literal>, and <literal>0</literal>
+ are still supported. The positive values emit logs for the
+ <literal>received</literal>, <literal>authenticated</literal>, and
+ <literal>authorized</literal> stages.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..066bc5254ee 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -38,6 +38,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/ipc.h"
+#include "tcop/backend_startup.h"
#include "utils/memutils.h"
/*----------------------------------------------------------------
@@ -317,7 +318,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authenticated' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +351,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +635,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATED) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..b4f34c57a1b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,6 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
bool enable_bonjour = false;
char *bonjour_name;
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..84db47231f6 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -34,13 +34,17 @@
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
+#include "utils/varlena.h"
/* GUCs */
bool Trace_connection_negotiation = false;
+int log_connections = 0;
+char *log_connections_string = NULL;
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
@@ -48,6 +52,7 @@ static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void process_startup_packet_die(SIGNAL_ARGS);
static void StartupPacketTimeoutHandler(void);
+static int validate_log_connections_options(List *elemlist);
/*
* Entry point for a new backend process.
@@ -201,8 +206,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIVED)
{
if (remote_port[0])
ereport(LOG,
@@ -924,3 +929,146 @@ StartupPacketTimeoutHandler(void)
{
_exit(1);
}
+
+/*
+ * Helper for the log_connections GUC check hook.
+ *
+ * `elemlist` is the listified string input passed to check_log_connections().
+ * check_log_connections() is responsible for cleaning up `elemlist`.
+ *
+ * validate_log_connections_options() returns the flags that should be stored
+ * in the log_connections GUC by the assign_hook or -1 if the input is not
+ * valid and we should error out.
+ */
+static int
+validate_log_connections_options(List *elemlist)
+{
+ int flags = 0;
+ ListCell *l;
+ char *item;
+
+ /* For backwards compatibility, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", LOG_CONNECTION_NONE},
+ {"false", LOG_CONNECTION_NONE},
+ {"no", LOG_CONNECTION_NONE},
+ {"0", LOG_CONNECTION_NONE},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
+
+ /* If an empty string was passed, we're done */
+ if (list_length(elemlist) == 0)
+ return 0;
+
+ /*
+ * Now check for the backwards compatibility options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatibility option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
+ GUC_check_errdetail("Cannot specify log_connections option \"%s\" in a list with other options.",
+ item);
+ return -1;
+ }
+
+ return option.val;
+ }
+
+ /*
+ * Now check the stage-specific options. The empty string was already
+ * handled.
+ */
+ foreach(l, elemlist)
+ {
+ static const struct config_enum_entry options[] = {
+ {"received", LOG_CONNECTION_RECEIVED},
+ {"authenticated", LOG_CONNECTION_AUTHENTICATED},
+ {"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"all", LOG_CONNECTION_ALL},
+ };
+
+ item = lfirst(l);
+ for (size_t i = 0; i < lengthof(options); i++)
+ {
+ struct config_enum_entry option = options[i];
+
+ if (pg_strcasecmp(item, option.name) == 0)
+ {
+ flags |= option.val;
+ goto next;
+ }
+ }
+
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ return -1;
+
+next: ;
+ }
+
+ return flags;
+}
+
+
+/*
+ * GUC check hook for log_connections
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ int flags;
+ char *rawstring;
+ List *elemlist;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitIdentifierString(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Validation logic is all in the helper */
+ flags = validate_log_connections_options(elemlist);
+
+ /* Time for cleanup */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ /* We didn't succeed */
+ if (flags == -1)
+ return false;
+
+ /*
+ * We succeeded, so allocate `extra` and save the flags there for use by
+ * assign_log_connections.
+ */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+}
+
+/*
+ * GUC assign hook for log_connections
+ */
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..dc6484bb092 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -54,6 +54,7 @@
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -252,7 +253,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..71070465379 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4886,6 +4877,18 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs information about events during connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..aca610f4fad 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections=all". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,7 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = ''
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..aa786bfacf3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,6 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..66d370c1e00 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -16,6 +16,8 @@
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
+extern PGDLLIMPORT int log_connections;
+extern PGDLLIMPORT char *log_connections_string;
/*
* CAC_state is passed from postmaster to the backend process, to indicate
@@ -39,6 +41,33 @@ typedef struct BackendStartupData
CAC_state canAcceptConnections;
} BackendStartupData;
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ *
+ * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
+ * establishment and setup where we may emit a log message.
+ *
+ * ALL is a convenience alias for when all stages should be logged.
+ *
+ * NONE is an alias for when no connection logging should be done.
+ *
+ * ON is backwards compatibility alias for the connection events that were
+ * logged in Postgres versions < 18.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_NONE = 0,
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIVED |
+ LOG_CONNECTION_AUTHENTICATED |
+ LOG_CONNECTION_AUTHORIZED,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
+} LogConnectionOption;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/test/postmaster/t/002_connection_limits.pl b/src/test/postmaster/t/002_connection_limits.pl
index 8cfa6e0ced5..f19d6e73ac9 100644
--- a/src/test/postmaster/t/002_connection_limits.pl
+++ b/src/test/postmaster/t/002_connection_limits.pl
@@ -19,9 +19,57 @@ $node->init(
$node->append_conf('postgresql.conf', "max_connections = 6");
$node->append_conf('postgresql.conf', "reserved_connections = 2");
$node->append_conf('postgresql.conf', "superuser_reserved_connections = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
$node->start;
+# Before testing connection limits, test the behavior of log_connections
+
+$node->connect_ok("",
+ "no connection logging emitted when log_connections is the default (off)",
+ log_unlike => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized/,
+ ],);
+
+$node->safe_psql(
+ 'postgres',
+ q[ALTER SYSTEM SET log_connections = received,authorized;
+ SELECT pg_reload_conf();]);
+
+$node->connect_ok("",
+ "log_connections with subset of specified stages logs only those stages",
+ log_like => [
+ qr/connection received/,
+ qr/connection authorized/,
+ ],
+ log_unlike => [
+ qr/connection authenticated/,
+ ],);
+
+$node->safe_psql(
+ 'postgres',
+ q[ALTER SYSTEM SET log_connections = on; SELECT pg_reload_conf();]);
+
+$node->connect_ok("",
+ "log_connections 'on' works as expected for backwards compatibility",
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized/,
+ ],);
+
+$node->safe_psql(
+ 'postgres',
+ q[ALTER SYSTEM SET log_connections = 'all'; SELECT pg_reload_conf();]);
+
+$node->connect_ok("",
+ "all log connections messages printed when 'all' specified",
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized/,
+ ],);
+
$node->safe_psql(
'postgres', qq{
CREATE USER regress_regular LOGIN;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..29e52b64994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1555,6 +1555,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
Hi,
On 2025-01-20 15:01:38 +0000, Bertrand Drouvot wrote:
Regarding the TimestampTz vs instr_time choice, we have things like:
+ TimestampTz fork_time = ((BackendStartupData *) startup_data)->fork_time; + TimestampTz cur_time = GetCurrentTimestamp(); + + conn_timing.fork_duration = TimestampDifferenceMicroseconds(fork_time, + cur_time);but looking at TimestampDifferenceMicroseconds():
"
/* If start_time is in the future or now, no time has elapsed */
if (start_time >= stop_time)
return 0;
"I think that it can happen due to time changes.
It shouldn't during proper operation, the time should be accellerated or
slowed down, but not have backwards jumps.
So with TimestampTz, we would:
1. return 0 if we moved the time backward
2. provide an inflated duration including the time jump (if the time move
forward).But with instr_time (and on systems that support CLOCK_MONOTONIC) then
pg_clock_gettime_ns() should not be affected by system time change IIUC.
It still does jump around a bit on some systems, even if it shouldn't. It's
not at all rare to see time distontinuities when getting scheduled on another
socket than before or when a VM got migrated. It shouldn't happen, but does.
Though time changes are "rare", given the fact that those metrics could provide
"inaccurate" measurements during that particular moment (time change) then it
might be worth considering instr_time instead for this particular metric.
We calculate the times for log_min_duration_statement etc via
GetCurrentTimestamp(), I don't think it's worth worrying about that here.
I think it'd be better to use absolute times and store them as such in
ConnectionTimes or whatever. That way we have information about when a
connection was established for some future SQL functions and for debugging
problems.
Greetings,
Andres Freund
On Thu, Mar 6, 2025 at 2:10 PM Andres Freund <andres@anarazel.de> wrote:
I think it'd be better to use absolute times and store them as such in
ConnectionTimes or whatever. That way we have information about when a
connection was established for some future SQL functions and for debugging
problems.
Attached v12 does this (uses timestamps instead of instr_time).
I've done some assorted cleanup. Most notably:
I removed LOG_CONNECTION_NONE because it could lead to wrong results
to have a bitmask with a flag value that is 0 (it could be set at the
same time other flags are set, so you could never use it correctly).
I updated the tests and moved them from the postmaster test suite to
the authentication test suite. They seemed like a slightly better fit
there.
I changed the user facing references to ready-for-query to
ready-to-use (it seemed less confusing).
I changed the process creation duration in the EXEC_BACKEND case to
_not_ include the overhead of copying data from shared memory and
reloading GUCs and the other setup done to get the backend in the
right state. Jacob at some point had asked about this, and I didn't
have a satisfactory answer. I'm not really sure what would be more
useful to end users.
- Melanie
Attachments:
v12-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v12-0002-Add-connection-establishment-duration-logging.patchDownload
From 277c32f09c35b688f0c415c758ad8d1203f2fe06 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:05:19 -0500
Subject: [PATCH v12 2/2] Add connection establishment duration logging
Add log_connections option 'ready_for_use' which logs durations of
several key parts of connection establishment and setup the first time
the backend is ready for query.
For an incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. This logging provides visibility
into authentication and fork duration as well as the end-to-end
connection establishment and initialization time.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData instead of simply saved in backend local memory.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 9 ++++
src/backend/postmaster/launch_backend.c | 35 ++++++++++++++
src/backend/postmaster/postmaster.c | 6 +++
src/backend/tcop/backend_startup.c | 4 ++
src/backend/tcop/postgres.c | 35 ++++++++++++++
src/backend/utils/init/postinit.c | 6 +++
src/include/miscadmin.h | 7 +++
src/include/tcop/backend_startup.h | 56 +++++++++++++++++++++--
src/include/utils/timestamp.h | 9 ++++
src/test/authentication/t/001_password.pl | 15 +++++-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 179 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fcfa7349e48..b5288132063 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7362,6 +7362,15 @@ local0.* /var/log/postgresql
</entry>
</row>
+ <row>
+ <entry><literal>ready_for_use</literal></entry>
+ <entry>
+ Logs the duration of connection establishment and setup as well as
+ several establishment component durations. At this point, the
+ backend is ready for use.
+ </entry>
+ </row>
+
<row>
<entry><literal>all</literal></entry>
<entry>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..44c8705424d 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates process creation for logging */
+ if (IsConnectionBackend(child_type))
+ ((BackendStartupData *) startup_data)->fork_started = GetCurrentTimestamp();
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Capture and transfer timings that may be needed for logging */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = GetCurrentTimestamp();
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,11 +600,19 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ TimestampTz fork_end;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
whereToSendOutput = DestNone;
+ /*
+ * Capture the end of process creation for logging. We don't include the
+ * time spent copying data from shared memory and setting up the backend
+ * as a proper postmaster child.
+ */
+ fork_end = GetCurrentTimestamp();
+
/* Setup essential subsystems (to ensure elog() behaves sanely) */
InitializeGUCOptions();
@@ -615,6 +637,9 @@ SubPostmasterMain(int argc, char *argv[])
if (!found)
elog(ERROR, "unknown child kind %s", child_kind);
+ if (IsConnectionBackend(child_type))
+ conn_timing.fork_end = GetCurrentTimestamp();
+
/* Read in the variables file */
read_backend_variables(argv[2], &startup_data, &startup_data_len);
@@ -648,6 +673,16 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /* Capture and transfer timings that may be needed for log_connections */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = fork_end;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b4f34c57a1b..57155c00e01 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -3477,6 +3477,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment and setup total duration).
+ */
+ startup_data.socket_created = GetCurrentTimestamp();
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c56dc1a367e..b4263d6f5cb 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -46,6 +46,9 @@ bool Trace_connection_negotiation = false;
uint32 log_connections = 0;
char *log_connections_string = NULL;
+/* Other globals */
+ConnectionTiming conn_timing = {0};
+
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
@@ -1002,6 +1005,7 @@ validate_log_connections_options(List *elemlist, uint32 *flags)
{"received", LOG_CONNECTION_RECEIVED},
{"authenticated", LOG_CONNECTION_AUTHENTICATED},
{"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"ready_for_use", LOG_CONNECTION_READY_FOR_USE},
{"all", LOG_CONNECTION_ALL},
};
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..801e9e11fff 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -66,6 +66,7 @@
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinval.h"
+#include "tcop/backend_startup.h"
#include "tcop/fastpath.h"
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
@@ -4153,6 +4154,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4609,39 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment.
+ */
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_READY_FOR_USE) &&
+ IsConnectionBackend(MyBackendType))
+ {
+ uint64 total_duration,
+ fork_duration,
+ auth_duration;
+
+ conn_timing.ready_for_use = GetCurrentTimestamp();
+
+ total_duration =
+ TimestampDifferenceMicroseconds(conn_timing.socket_create,
+ conn_timing.ready_for_use);
+ fork_duration =
+ TimestampDifferenceMicroseconds(conn_timing.fork_start,
+ conn_timing.fork_end);
+ auth_duration =
+ TimestampDifferenceMicroseconds(conn_timing.auth_start,
+ conn_timing.auth_end);
+
+ ereport(LOG,
+ errmsg("connection ready for use: time spent (ms): total=%.3f, process creation=%.3f, authentication=%.3f",
+ (double) total_duration / NS_PER_US,
+ (double) fork_duration / NS_PER_US,
+ (double) auth_duration / NS_PER_US));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index dc6484bb092..9e7c2b8f02f 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -195,6 +195,9 @@ PerformAuthentication(Port *port)
/* This should be set already, but let's make sure */
ClientAuthInProgress = true; /* limit visibility of log messages */
+ /* Capture authentication start time for logging */
+ conn_timing.auth_start = GetCurrentTimestamp();
+
/*
* In EXEC_BACKEND case, we didn't inherit the contents of pg_hba.conf
* etcetera from the postmaster, and have to load them ourselves.
@@ -253,6 +256,9 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Capture authentication end time for logging */
+ conn_timing.auth_end = GetCurrentTimestamp();
+
if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..6deffd0a202 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -394,6 +394,13 @@ extern PGDLLIMPORT BackendType MyBackendType;
(AmAutoVacuumLauncherProcess() || \
AmLogicalSlotSyncWorkerProcess())
+/*
+ * Backend types that are spawned by the postmaster to serve a client
+ * connection.
+ */
+#define IsConnectionBackend(backend_type) \
+ (backend_type == B_BACKEND || backend_type == B_WAL_SENDER)
+
extern const char *GetBackendTypeDesc(BackendType backendType);
extern void SetDatabasePath(const char *path);
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 7630f0907a0..8216a0a6a74 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,11 +14,16 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "utils/timestamp.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
extern PGDLLIMPORT uint32 log_connections;
extern PGDLLIMPORT char *log_connections_string;
+/* Other globals */
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
/*
* CAC_state is passed from postmaster to the backend process, to indicate
* whether the connection should be accepted, or if the process should just
@@ -39,13 +44,26 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ TimestampTz socket_created;
+
+ /*
+ * Time at which the postmaster initiates process creation -- either
+ * through fork or otherwise. Only used for client and wal sender
+ * connections.
+ */
+ TimestampTz fork_started;
} BackendStartupData;
/*
* Granular control over which messages to log for the log_connections GUC.
*
- * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
- * establishment and setup where we may emit a log message.
+ * RECEIVED, AUTHENTICATED, AUTHORIZED, and READY_FOR_USE are different stages
+ * of connection establishment and setup where we may emit a log message.
*
* ALL is a convenience alias for when all stages should be logged.
*
@@ -59,14 +77,46 @@ typedef enum LogConnectionOption
LOG_CONNECTION_RECEIVED = (1 << 0),
LOG_CONNECTION_AUTHENTICATED = (1 << 1),
LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_READY_FOR_USE = (1 << 3),
LOG_CONNECTION_ON =
LOG_CONNECTION_RECEIVED |
LOG_CONNECTION_AUTHENTICATED |
LOG_CONNECTION_AUTHORIZED,
LOG_CONNECTION_ALL =
- LOG_CONNECTION_ON,
+ LOG_CONNECTION_ON |
+ LOG_CONNECTION_READY_FOR_USE,
} LogConnectionOption;
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders.
+ *
+ * Used to emit log messages according to the options specified in the
+ * log_connections GUC.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created and the time at which
+ * the connection is fully set up and first ready for query. Together
+ * these represent the total connection establishment and setup time.
+ */
+ TimestampTz socket_create;
+ TimestampTz ready_for_use;
+
+ /* Time at which process creation was initiated */
+ TimestampTz fork_start;
+
+ /* Time at which process creation was completed */
+ TimestampTz fork_end;
+
+ /* Time at which authentication started */
+ TimestampTz auth_start;
+
+ /* Time at which authentication was finished */
+ TimestampTz auth_end;
+} ConnectionTiming;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 9963bddc0ec..8c205859c3b 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -85,6 +85,15 @@ IntervalPGetDatum(const Interval *X)
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+/* Helper for simple subtraction between two timestamps */
+static inline uint64
+TimestampDifferenceMicroseconds(TimestampTz start_time,
+ TimestampTz stop_time)
+{
+ if (start_time >= stop_time)
+ return 0;
+ return (uint64) stop_time - start_time;
+}
/* Set at postmaster start */
extern PGDLLIMPORT TimestampTz PgStartTime;
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index e2a201ebb3f..33cf14e6529 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -68,8 +68,19 @@ $node->start;
# Test behavior of log_connections GUC
+$node->connect_ok("",
+ "log_connections 'on' works as expected for backwards compatibility",
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized/,
+ ],
+ log_unlike => [
+ qr/connection ready for use/,
+ ],);
+
$node->safe_psql('postgres',
- q[ALTER SYSTEM SET log_connections = received,authorized;
+ q[ALTER SYSTEM SET log_connections = received,authorized,ready_for_use;
SELECT pg_reload_conf();]);
$node->connect_ok("",
@@ -77,6 +88,7 @@ $node->connect_ok("",
log_like => [
qr/connection received/,
qr/connection authorized/,
+ qr/connection ready for use/,
],
log_unlike => [
qr/connection authenticated/,
@@ -91,6 +103,7 @@ $node->connect_ok("",
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized/,
+ qr/connection ready for use/,
],);
# could fail in FIPS mode
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 29e52b64994..3da74999634 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
v12-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v12-0001-Modularize-log_connections-output.patchDownload
From 6c4360a06181628c3b8436290b219ec2d2a44510 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:02:45 -0500
Subject: [PATCH v12 1/2] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC of the
connection stages to log.
The current log_connections options are 'received', 'authenticated', and
'authorized'. The empty string disables all connection logging. 'all'
enables all available connection logging.
This gives users more control over the volume of connection logging.
For backwards compatibility, the most common values for the
log_connections boolean are still supported ('on', 'off', 1, 0, true,
false).
This patch has much of the same behavior as [1] but was developed
independently.
[1] https://postgr.es/m/flat/CAA8Fd-qCB96uwfgMKrzfNs32mqqysi53yZFNVaRNJ6xDthZEgA%40mail.gmail.com
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 70 +++++++-
src/backend/libpq/auth.c | 9 +-
src/backend/postmaster/postmaster.c | 1 -
src/backend/tcop/backend_startup.c | 157 +++++++++++++++++-
src/backend/utils/init/postinit.c | 3 +-
src/backend/utils/misc/guc_tables.c | 21 ++-
src/backend/utils/misc/postgresql.conf.sample | 4 +-
src/include/postmaster/postmaster.h | 1 -
src/include/tcop/backend_startup.h | 28 ++++
src/include/utils/guc_hooks.h | 2 +
src/test/authentication/t/001_password.pl | 27 +++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 300 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..fcfa7349e48 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,80 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes the specified stages of each connection attempt to the server to
+ be logged. The default is the empty string, <literal>''</literal>,
+ which disables all connection logging. The following are the options
+ that may be specified:
+ </para>
+
+ <table id="log-connections-stages">
+ <title>Log Connection Stages</title>
+ <tgroup cols="2">
+ <colspec colname="col1" colwidth="1*"/>
+ <colspec colname="col2" colwidth="2*"/>
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>received</literal></entry>
+ <entry>Logs receipt of a connection.</entry>
+ </row>
+
+ <row>
+ <entry><literal>authenticated</literal></entry>
+ <entry>
+ Logs the original identity used by an authentication method to
+ identify a user. Failed authentication is always logged regardless
+ of the value of this setting.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>authorized</literal></entry>
+ <entry>
+ Logs successful completion of authorization. At this point the
+ connection has been fully established.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>all</literal></entry>
+ <entry>
+ A convenience alias to log all connection stages. If
+ <literal>all</literal> is specified in a list of other options, all
+ connection logging will be enabled.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ For the purposes of backwards compatibility, <literal>on</literal>,
+ <literal>off</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>yes</literal>,
+ <literal>no</literal>, <literal>1</literal>, and <literal>0</literal>
+ are still supported. The positive values emit logs for the
+ <literal>received</literal>, <literal>authenticated</literal>, and
+ <literal>authorized</literal> stages.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..066bc5254ee 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -38,6 +38,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/ipc.h"
+#include "tcop/backend_startup.h"
#include "utils/memutils.h"
/*----------------------------------------------------------------
@@ -317,7 +318,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authenticated' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +351,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATED)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +635,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATED) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..b4f34c57a1b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,6 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
bool enable_bonjour = false;
char *bonjour_name;
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..c56dc1a367e 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -34,13 +34,17 @@
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
+#include "utils/varlena.h"
/* GUCs */
bool Trace_connection_negotiation = false;
+uint32 log_connections = 0;
+char *log_connections_string = NULL;
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
@@ -48,6 +52,7 @@ static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void process_startup_packet_die(SIGNAL_ARGS);
static void StartupPacketTimeoutHandler(void);
+static bool validate_log_connections_options(List *elemlist, uint32 *flags);
/*
* Entry point for a new backend process.
@@ -201,8 +206,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIVED)
{
if (remote_port[0])
ereport(LOG,
@@ -924,3 +929,151 @@ StartupPacketTimeoutHandler(void)
{
_exit(1);
}
+
+/*
+ * Helper for the log_connections GUC check hook.
+ *
+ * `elemlist` is a listified version of the string input passed to the
+ * log_connections GUC check hook, check_log_connections().
+ * check_log_connections() is responsible for cleaning up `elemlist`.
+ *
+ * validate_log_connections_options() returns false if an error was
+ * encountered and the GUC input could not be validated and true otherwise.
+ *
+ * `flags` returns the flags that should be stored in the log_connections GUC
+ * by its assign hook.
+ */
+static bool
+validate_log_connections_options(List *elemlist, uint32 *flags)
+{
+ ListCell *l;
+ char *item;
+
+ /* For backwards compatibility, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
+
+ *flags = 0;
+
+ /* If an empty string was passed, we're done */
+ if (list_length(elemlist) == 0)
+ return true;
+
+ /*
+ * Now check for the backwards compatibility options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatibility option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
+ GUC_check_errdetail("Cannot specify log_connections option \"%s\" in a list with other options.",
+ item);
+ return false;
+ }
+
+ *flags = option.val;
+ return true;
+ }
+
+ /*
+ * Now check the stage-specific options. The empty string was already
+ * handled.
+ */
+ foreach(l, elemlist)
+ {
+ static const struct config_enum_entry options[] = {
+ {"received", LOG_CONNECTION_RECEIVED},
+ {"authenticated", LOG_CONNECTION_AUTHENTICATED},
+ {"authorized", LOG_CONNECTION_AUTHORIZED},
+ {"all", LOG_CONNECTION_ALL},
+ };
+
+ item = lfirst(l);
+ for (size_t i = 0; i < lengthof(options); i++)
+ {
+ struct config_enum_entry option = options[i];
+
+ if (pg_strcasecmp(item, option.name) == 0)
+ {
+ *flags |= option.val;
+ goto next;
+ }
+ }
+
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ return false;
+
+next: ;
+ }
+
+ return true;
+}
+
+
+/*
+ * GUC check hook for log_connections
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ uint32 flags;
+ char *rawstring;
+ List *elemlist;
+ bool success;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitIdentifierString(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Validation logic is all in the helper */
+ success = validate_log_connections_options(elemlist, &flags);
+
+ /* Time for cleanup */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ if (!success)
+ return false;
+
+ /*
+ * We succeeded, so allocate `extra` and save the flags there for use by
+ * assign_log_connections().
+ */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+}
+
+/*
+ * GUC assign hook for log_connections
+ */
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..dc6484bb092 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -54,6 +54,7 @@
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -252,7 +253,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZED)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..350ead95993 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4886,6 +4877,18 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs connection establishment stages."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..aca610f4fad 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections=all". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,7 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = ''
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..aa786bfacf3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,6 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..7630f0907a0 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -16,6 +16,8 @@
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
+extern PGDLLIMPORT uint32 log_connections;
+extern PGDLLIMPORT char *log_connections_string;
/*
* CAC_state is passed from postmaster to the backend process, to indicate
@@ -39,6 +41,32 @@ typedef struct BackendStartupData
CAC_state canAcceptConnections;
} BackendStartupData;
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ *
+ * RECEIVED, AUTHENTICATED, and AUTHORIZED are different stages of connection
+ * establishment and setup where we may emit a log message.
+ *
+ * ALL is a convenience alias for when all stages should be logged.
+ *
+ * NONE is an alias for when no connection logging should be done.
+ *
+ * ON is backwards compatibility alias for the connection stages that were
+ * logged in Postgres versions < 18.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
+ LOG_CONNECTION_AUTHORIZED = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIVED |
+ LOG_CONNECTION_AUTHENTICATED |
+ LOG_CONNECTION_AUTHORIZED,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
+} LogConnectionOption;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 4ce22ccbdf2..e2a201ebb3f 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -66,6 +66,33 @@ $node->init;
$node->append_conf('postgresql.conf', "log_connections = on\n");
$node->start;
+# Test behavior of log_connections GUC
+
+$node->safe_psql('postgres',
+ q[ALTER SYSTEM SET log_connections = received,authorized;
+ SELECT pg_reload_conf();]);
+
+$node->connect_ok("",
+ "log_connections with subset of specified stages logs only those stages",
+ log_like => [
+ qr/connection received/,
+ qr/connection authorized/,
+ ],
+ log_unlike => [
+ qr/connection authenticated/,
+ ],);
+
+$node->safe_psql('postgres',
+ q[ALTER SYSTEM SET log_connections = 'all'; SELECT pg_reload_conf();]);
+
+$node->connect_ok("",
+ "log_connections 'all' logs all connection stages",
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized/,
+ ],);
+
# could fail in FIPS mode
my $md5_works = ($node->psql('postgres', "select md5('')") == 0);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..29e52b64994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1555,6 +1555,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
On Thu, Mar 6, 2025 at 3:16 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
Jacob at some point had asked about this, and I didn't
have a satisfactory answer. I'm not really sure what would be more
useful to end users.
For the record, I'm not really sure either. I don't have a strong
opinion either way.
--Jacob
On 2025/03/07 8:16, Melanie Plageman wrote:
On Thu, Mar 6, 2025 at 2:10 PM Andres Freund <andres@anarazel.de> wrote:
I think it'd be better to use absolute times and store them as such in
ConnectionTimes or whatever. That way we have information about when a
connection was established for some future SQL functions and for debugging
problems.Attached v12 does this (uses timestamps instead of instr_time).
Thanks for updating the patch!
Here are the comments for v1:
With the patch, any unambiguous prefix of a valid boolean value,
like 'y', is no longer accepted even though it's currently valid
for boolean GUCs. I don’t have a strong opinion on whether
we should maintain compatibility for this, but I wanted to
mention it for the record. If we do, we might need to use
parse_bool() to interpret the values.
+ * NONE is an alias for when no connection logging should be done.
The comment about "NONE" should be removed since LOG_CONNECTION_NONE was deleted.
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIVED = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATED = (1 << 1),
Since the GUC is named log_connections, I'm tempted to name
LOG_CONNECTIONS_XXX and LogConnectionsOption here instead.
This is just a suggestion. If others prefer the current names,
I’m fine with that.
Comments for v2:
+ if (IsConnectionBackend(child_type))
+ conn_timing.fork_end = GetCurrentTimestamp();
In SubPostmasterMain(), GetCurrentTimestamp() is called twice to
set fork_end. It seems unnecessary. We can remove the above call?
+ /* Capture authentication start time for logging */
+ conn_timing.auth_start = GetCurrentTimestamp();
In the EXEC_BACKEND case, the authentication start time differs
from when the authentication timeout begins. But shouldn't these
be the same?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Hi,
On Thu, Mar 06, 2025 at 02:10:47PM -0500, Andres Freund wrote:
On 2025-01-20 15:01:38 +0000, Bertrand Drouvot wrote:
/* If start_time is in the future or now, no time has elapsed */
if (start_time >= stop_time)
return 0;
"I think that it can happen due to time changes.
So with TimestampTz, we would:
1. return 0 if we moved the time backward
2. provide an inflated duration including the time jump (if the time move
forward).But with instr_time (and on systems that support CLOCK_MONOTONIC) then
pg_clock_gettime_ns() should not be affected by system time change IIUC.It still does jump around a bit on some systems, even if it shouldn't. It's
not at all rare to see time distontinuities when getting scheduled on another
socket than before
Interesting, yeah I can imagine that could happen.
or when a VM got migrated. It shouldn't happen, but does.
Agree, those are convincing examples.
I think it'd be better to use absolute times and store them as such in
ConnectionTimes or whatever.
It was still not clear to me why using TimestampTz would be better, until I read:
That way we have information about when a
connection was established for some future SQL functions and for debugging
problems.
Thanks!
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Thu, Mar 06, 2025 at 11:41:03AM -0500, Melanie Plageman wrote:
I still need to do some more manual testing and validation.
On Thu, Mar 6, 2025 at 9:56 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:+ /* If an empty string was passed, we're done. */ + if (list_length(elemlist) == 0) + return 0; +< -- snip -->
what about storing the list_length(elemlist) at the start to avoid the multiple
list_length(elemlist) calls?Since it would only be called twice
Oh right, I was focusing on the top loop but missed the "return" if the length is
1.
and it has the length stored in
the List struct, I prefer it as is -- without an extra local variable.
Yeah, me too, sorry for the noise.
How do you find spelling mistakes in patches usually?
Given the fact that I'm not a native english speaker I would say: By luck ;-)
I just try do one pass during the review just focusing on those.
I tried `git
show | aspell list` -- but of course that includes lots of variables
and such that aren't actually spelling mistakes.
Trying to "automate" is actually a good idea!
Well, I actually didn't want to call it "timings" because it implies
that we only measure the timings when it is specified. But we actually
get the timings unconditionally for client backends and wal sender.
Don't you think it is more confusing for the user if we call it
timings and users think if they don't include that timings aren't
measured?
That's a good question. I think the expectations are set in the log_connections
GUC documentation. It says "Causes the specified stages of each connection attempt
to the server to be logged". So for me that would be like: log yes or no the
timings.
Of course to be logged those need to be measured and one could expect the timings
not to be measured if timings is not set.
But at the end, what we're "really" interested in this thread, given its $SUBJECT,
is to actually log the timings.
So yeah, from my point of view I think that it would be better if the option
name highlights the fact that it is about timing (and not only in its description
as with ready_for_query). One option to address your concern could be to re-word
the doc a bit saying (logs timings that are measured independently of the GUC value,
something like this).
That said If you feel that it's too confusing to use "timings" then I'm fine too.
Also, I thought changing the message output to say "ready for query"
somewhere in it makes it more clear when that message is actually
emitted in a connection's lifetime. What do you think?
I agree if we keep ready_for_query as the option name.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Hi,
On Thu, Mar 06, 2025 at 06:16:07PM -0500, Melanie Plageman wrote:
Attached v12 does this (uses timestamps instead of instr_time).
Thanks for the new version!
I've done some assorted cleanup. Most notably:
I removed LOG_CONNECTION_NONE because it could lead to wrong results
to have a bitmask with a flag value that is 0 (it could be set at the
same time other flags are set, so you could never use it correctly).
Oh right, and starting with LOG_CONNECTION_NONE/OFF = (1 << 0) does not
make that much sense...
A couple of comments regarding 0002:
=== 01
Given that conn_timing.ready_for_use is only used here:
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_READY_FOR_USE) &&
+ IsConnectionBackend(MyBackendType))
+ {
+ uint64 total_duration,
+ fork_duration,
+ auth_duration;
+
+ conn_timing.ready_for_use = GetCurrentTimestamp();
+
+ total_duration =
+ TimestampDifferenceMicroseconds(conn_timing.socket_create,
+ conn_timing.ready_for_use);
I wonder if ready_for_use needs to be part of ConnectionTiming after all.
I mean we could just:
"
total_duration = TimestampDifferenceMicroseconds(conn_timing.socket_create, GetCurrentTimestamp())
"
That said, having ready_for_use part of ConnectionTiming could be
useful if we decide to create a SQL API on top of this struct though. So,
that's probably fine and better as it is.
And if we keep it part of ConnectionTiming, then:
=== 02
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_READY_FOR_USE) &&
+ IsConnectionBackend(MyBackendType))
What about getting rid of reported_first_ready_for_query and check if
conn_timing.ready_for_use is zero instead?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Fri, Mar 7, 2025 at 6:16 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
On Thu, Mar 06, 2025 at 11:41:03AM -0500, Melanie Plageman wrote:
Well, I actually didn't want to call it "timings" because it implies
that we only measure the timings when it is specified. But we actually
get the timings unconditionally for client backends and wal sender.Don't you think it is more confusing for the user if we call it
timings and users think if they don't include that timings aren't
measured?That's a good question. I think the expectations are set in the log_connections
GUC documentation. It says "Causes the specified stages of each connection attempt
to the server to be logged". So for me that would be like: log yes or no the
timings.Of course to be logged those need to be measured and one could expect the timings
not to be measured if timings is not set.But at the end, what we're "really" interested in this thread, given its $SUBJECT,
is to actually log the timings.
I thought about this more. Thanks for taking the time to discuss.
For one, I take your point: I originally proposed the patch because we
wanted some visibility into how long various stages of connection
setup took -- not because we wanted to log when the backend was ready
for query. I'm not sure that people would enable this option much if
it was called ready_for_use, since just logging when we are ready for
query is likely not that valuable.
This got me thinking more about the existing log connections messages
and whether or not they are actually all "stages". The authenticated
and authorized messages occur at nearly the same time in the
connection establishment timeline. So it seems like they aren't really
distinct stages. The messages contain different information and serve
different purposes. Looking at 9afffcb833d, which added the
authentication message, it gives the example of a database user with a
different authentication identity. The authentication message has the
auth id and the auth method. The authorized message has the db and db
username.
So, I would call "received" and "authorized" stages and the
authentication ID message not a stage. I think I should not call these
"log_connections stages" in the docs and comments and instead call
them "log_connections options". That also makes me wonder if the
"authenticated" log_connections option should actually be called
"auth_id" or something similar.
And I am wondering if the "timings" option should be called "timings"
or "durations"? I want to convey that it is about printing connection
setup durations and not about whether or not we measure timings. But
log_connections=durations sounds more like it logs the total duration
of the connection setup and not component parts...
<-- snip-->
Also, I thought changing the message output to say "ready for query"
somewhere in it makes it more clear when that message is actually
emitted in a connection's lifetime. What do you think?I agree if we keep ready_for_query as the option name.
I actually think even if we change the option name, it could be
valuable to have the message begin with "connection ready for use";
otherwise, it's unclear when we are printing the message. Logging
messages are very tied to what was happening when they were emitted.
Usually they include some context on when they were emitted. Thus, I
think it makes sense to somehow contextualize the message in this way
in the text. It does make the message rather long, though...
- Melanie
On Fri, Mar 7, 2025 at 12:29 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
This got me thinking more about the existing log connections messages
and whether or not they are actually all "stages". The authenticated
and authorized messages occur at nearly the same time in the
connection establishment timeline. So it seems like they aren't really
distinct stages. The messages contain different information and serve
different purposes. Looking at 9afffcb833d, which added the
authentication message, it gives the example of a database user with a
different authentication identity. The authentication message has the
auth id and the auth method. The authorized message has the db and db
username.So, I would call "received" and "authorized" stages and the
authentication ID message not a stage. I think I should not call these
"log_connections stages" in the docs and comments and instead call
them "log_connections options".
I think it's a valid point.
If I add a hypothetical auth method in the future that authenticates,
and then farms the authorization decision out to some slow-moving
network machinery, would "authenticated" retroactively become a stage
then? (OAuth almost does this today... but it's not quite separated
enough for me to claim it as an example.) If we call them "options"
instead, I guess we don't have to worry about shifting internals.
That also makes me wonder if the
"authenticated" log_connections option should actually be called
"auth_id" or something similar.
To bikeshed the specific suggestion of "auth_id": both Peter E and
Robert have previously expressed concern that my internal name choice
of "authn_id" was too opaque, and that maybe I should have just
expanded the terms. Also, I think if one option is called
"authorized", the other half should probably be called "authenticated"
if for no other reason than symmetry. It also matches the prefix used
in the logs, for English builds.
--Jacob
Thanks for the review!
On Fri, Mar 7, 2025 at 2:08 AM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
With the patch, any unambiguous prefix of a valid boolean value,
like 'y', is no longer accepted even though it's currently valid
for boolean GUCs. I don’t have a strong opinion on whether
we should maintain compatibility for this, but I wanted to
mention it for the record. If we do, we might need to use
parse_bool() to interpret the values.
Indeed. In fact, in master, any substring of true/yes/false/no would
be accepted. So I could set log_connections = tr;
It's true that with this patch only on,off,true,false,1,0,yes,no would
be supported. But I think that is okay. If we allow arbitrary
unambiguous substrings of the previously accepted values, it limits
what future log_connections options we can add (they couldn't clash
with any of those substrings or introduce ambiguity). I think it's
better we just draw a line now.
There is one complication, the boolean GUC infrastructure parses those
inputs but then the output is always "on" or "off" in this case. So,
on master if you set log_connections=1 and then do "show
log_connections" it will display "on". With this patch it would
display 1. We may be able to solve that with a show hook, but my
instinct is that that is not worth it. What do you think?
+ * NONE is an alias for when no connection logging should be done.
The comment about "NONE" should be removed since LOG_CONNECTION_NONE was deleted.
Thanks! Fixed in attached v13.
+typedef enum LogConnectionOption +{ + LOG_CONNECTION_RECEIVED = (1 << 0), + LOG_CONNECTION_AUTHENTICATED = (1 << 1),Since the GUC is named log_connections, I'm tempted to name
LOG_CONNECTIONS_XXX and LogConnectionsOption here instead.
This is just a suggestion. If others prefer the current names,
I’m fine with that.
For the enum and its values, I prefer the singular. log_connections is
about logging all connections. However, when we use these enums, it is
about logging a particular connection, so the singular feels more
appropriate.
Comments for v2:
+ if (IsConnectionBackend(child_type)) + conn_timing.fork_end = GetCurrentTimestamp();In SubPostmasterMain(), GetCurrentTimestamp() is called twice to
set fork_end. It seems unnecessary. We can remove the above call?
Whoops! Thanks for catching this. Fixed in attached v12.
+ /* Capture authentication start time for logging */ + conn_timing.auth_start = GetCurrentTimestamp();In the EXEC_BACKEND case, the authentication start time differs
from when the authentication timeout begins. But shouldn't these
be the same?
Good point. At some point, I thought maybe it made sense to include
the EXEC_BACKEND-specific setup stuff in the authentication duration
since it is one of the overheads of doing authentication on an
EXEC_BACKEND backend. But, now I realize you are right -- the user
wants to know, if, for example, something weird is happening in a
library somewhere that is making authentication really slow -- not how
long it took to reload the HBA file. I've changed it as you suggested.
An unrelated note about the attached v13:
I changed the language from log_connections "stages" to "options" and
"aspects" depending on the context. I also changed the name of the
option introduced in 0002 to "durations".
I'm a bit torn about having the tests in authentication/001_password.
On the one hand, it doesn't make sense to make a separate test file
and initialize a new postgres just to test two or three options of one
GUC. On the other hand, these tests don't fit very well in any
existing test file I could find. Maybe it is fine that they are in a
file about testing password authentication...
- Melanie
Attachments:
v13-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v13-0002-Add-connection-establishment-duration-logging.patchDownload
From 91cfc397d2d559bce4b68157a5a69bdd7cd29084 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:05:19 -0500
Subject: [PATCH v13 2/2] Add connection establishment duration logging
Add log_connections option 'durations' which logs durations of
several key parts of connection establishment and backend setup.
For an incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. This logging provides visibility
into authentication and fork duration as well as the end-to-end
connection establishment and initialization time.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData instead of simply saved in backend local memory.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 10 ++++
src/backend/postmaster/launch_backend.c | 31 +++++++++++++
src/backend/postmaster/postmaster.c | 6 +++
src/backend/tcop/backend_startup.c | 4 ++
src/backend/tcop/postgres.c | 35 ++++++++++++++
src/backend/utils/init/postinit.c | 6 +++
src/include/miscadmin.h | 7 +++
src/include/tcop/backend_startup.h | 56 +++++++++++++++++++++--
src/include/utils/timestamp.h | 9 ++++
src/test/authentication/t/001_password.pl | 18 +++++++-
src/tools/pgindent/typedefs.list | 1 +
11 files changed, 179 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 52f5a115391..b7180ccfffb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7366,6 +7366,16 @@ local0.* /var/log/postgresql
</entry>
</row>
+ <row>
+ <entry><literal>durations</literal></entry>
+ <entry>
+ Logs the time spent establishing the connection and setting up the
+ backend at the time the connection is ready for use. The log
+ message includes the total setup duration as well as several
+ component durations, including process creation and authentication.
+ </entry>
+ </row>
+
<row>
<entry><literal>all</literal></entry>
<entry>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..74b0c33adaa 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates process creation for logging */
+ if (IsConnectionBackend(child_type))
+ ((BackendStartupData *) startup_data)->fork_started = GetCurrentTimestamp();
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Capture and transfer timings that may be needed for logging */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = GetCurrentTimestamp();
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,11 +600,18 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ TimestampTz fork_end;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
whereToSendOutput = DestNone;
+ /*
+ * Capture the end of process creation for logging. We don't include the
+ * time spent copying data from shared memory and setting up the backend.
+ */
+ fork_end = GetCurrentTimestamp();
+
/* Setup essential subsystems (to ensure elog() behaves sanely) */
InitializeGUCOptions();
@@ -648,6 +669,16 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /* Capture and transfer timings that may be needed for log_connections */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = fork_end;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b4f34c57a1b..57155c00e01 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -3477,6 +3477,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment and setup total duration).
+ */
+ startup_data.socket_created = GetCurrentTimestamp();
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index fb0baeb4a24..2dc838e7dc1 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -46,6 +46,9 @@ bool Trace_connection_negotiation = false;
uint32 log_connections = 0;
char *log_connections_string = NULL;
+/* Other globals */
+ConnectionTiming conn_timing = {0};
+
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
@@ -999,6 +1002,7 @@ validate_log_connections_options(List *elemlist, uint32 *flags)
{"receipt", LOG_CONNECTION_RECEIPT},
{"authentication", LOG_CONNECTION_AUTHENTICATION},
{"authorization", LOG_CONNECTION_AUTHORIZATION},
+ {"durations", LOG_CONNECTION_DURATIONS},
{"all", LOG_CONNECTION_ALL},
};
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..0f140f41d80 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -66,6 +66,7 @@
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinval.h"
+#include "tcop/backend_startup.h"
#include "tcop/fastpath.h"
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
@@ -4153,6 +4154,7 @@ PostgresMain(const char *dbname, const char *username)
volatile bool send_ready_for_query = true;
volatile bool idle_in_transaction_timeout_enabled = false;
volatile bool idle_session_timeout_enabled = false;
+ bool reported_first_ready_for_query = false;
Assert(dbname != NULL);
Assert(username != NULL);
@@ -4607,6 +4609,39 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment and setup.
+ */
+ if (!reported_first_ready_for_query &&
+ (log_connections & LOG_CONNECTION_DURATIONS) &&
+ IsConnectionBackend(MyBackendType))
+ {
+ uint64 total_duration,
+ fork_duration,
+ auth_duration;
+
+ conn_timing.ready_for_use = GetCurrentTimestamp();
+
+ total_duration =
+ TimestampDifferenceMicroseconds(conn_timing.socket_create,
+ conn_timing.ready_for_use);
+ fork_duration =
+ TimestampDifferenceMicroseconds(conn_timing.fork_start,
+ conn_timing.fork_end);
+ auth_duration =
+ TimestampDifferenceMicroseconds(conn_timing.auth_start,
+ conn_timing.auth_end);
+
+ ereport(LOG,
+ errmsg("connection ready: total=%.3f ms, fork=%.3f ms, authentication=%.3f ms",
+ (double) total_duration / NS_PER_US,
+ (double) fork_duration / NS_PER_US,
+ (double) auth_duration / NS_PER_US));
+
+ reported_first_ready_for_query = true;
+ }
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 6c207e17768..4b2faf1ba9d 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -235,6 +235,9 @@ PerformAuthentication(Port *port)
}
#endif
+ /* Capture authentication start time for logging */
+ conn_timing.auth_start = GetCurrentTimestamp();
+
/*
* Set up a timeout in case a buggy or malicious client fails to respond
* during authentication. Since we're inside a transaction and might do
@@ -253,6 +256,9 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Capture authentication end time for logging */
+ conn_timing.auth_end = GetCurrentTimestamp();
+
if (log_connections & LOG_CONNECTION_AUTHORIZATION)
{
StringInfoData logmsg;
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..312eb3f9529 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -394,6 +394,13 @@ extern PGDLLIMPORT BackendType MyBackendType;
(AmAutoVacuumLauncherProcess() || \
AmLogicalSlotSyncWorkerProcess())
+/*
+ * Backend types that are spawned by the postmaster to serve a client
+ * or replication connection.
+ */
+#define IsConnectionBackend(backend_type) \
+ (backend_type == B_BACKEND || backend_type == B_WAL_SENDER)
+
extern const char *GetBackendTypeDesc(BackendType backendType);
extern void SetDatabasePath(const char *path);
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index b5a33a2b0e1..acc01461b76 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,11 +14,16 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "utils/timestamp.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
extern PGDLLIMPORT uint32 log_connections;
extern PGDLLIMPORT char *log_connections_string;
+/* Other globals */
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
/*
* CAC_state is passed from postmaster to the backend process, to indicate
* whether the connection should be accepted, or if the process should just
@@ -39,13 +44,26 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ TimestampTz socket_created;
+
+ /*
+ * Time at which the postmaster initiates process creation -- either
+ * through fork or otherwise. Only used for client and wal sender
+ * connections.
+ */
+ TimestampTz fork_started;
} BackendStartupData;
/*
* Granular control over which messages to log for the log_connections GUC.
*
- * RECEIPT, AUTHENTICATION, and AUTHORIZATION are different aspects of
- * connection establishment and backend setup for which we may emit a log
+ * RECEIPT, AUTHENTICATION, AUTHORIZATION, and DURATIONS are different aspects
+ * of connection establishment and backend setup for which we may emit a log
* message.
*
* ALL is a convenience alias equivalent to all of the above aspects.
@@ -58,14 +76,46 @@ typedef enum LogConnectionOption
LOG_CONNECTION_RECEIPT = (1 << 0),
LOG_CONNECTION_AUTHENTICATION = (1 << 1),
LOG_CONNECTION_AUTHORIZATION = (1 << 2),
+ LOG_CONNECTION_DURATIONS = (1 << 3),
LOG_CONNECTION_ON =
LOG_CONNECTION_RECEIPT |
LOG_CONNECTION_AUTHENTICATION |
LOG_CONNECTION_AUTHORIZATION,
LOG_CONNECTION_ALL =
- LOG_CONNECTION_ON,
+ LOG_CONNECTION_ON |
+ LOG_CONNECTION_DURATIONS,
} LogConnectionOption;
+/*
+ * A collection of timings and durations of various stages of connection
+ * establishment and setup for client backends and WAL senders.
+ *
+ * Used to emit log messages according to the options specified in the
+ * log_connections GUC.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created and the time at which
+ * the connection is fully set up and first ready for query. Together
+ * these represent the total connection establishment and setup time.
+ */
+ TimestampTz socket_create;
+ TimestampTz ready_for_use;
+
+ /* Time at which process creation was initiated */
+ TimestampTz fork_start;
+
+ /* Time at which process creation was completed */
+ TimestampTz fork_end;
+
+ /* Time at which authentication started */
+ TimestampTz auth_start;
+
+ /* Time at which authentication was finished */
+ TimestampTz auth_end;
+} ConnectionTiming;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 9963bddc0ec..8c205859c3b 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -85,6 +85,15 @@ IntervalPGetDatum(const Interval *X)
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+/* Helper for simple subtraction between two timestamps */
+static inline uint64
+TimestampDifferenceMicroseconds(TimestampTz start_time,
+ TimestampTz stop_time)
+{
+ if (start_time >= stop_time)
+ return 0;
+ return (uint64) stop_time - start_time;
+}
/* Set at postmaster start */
extern PGDLLIMPORT TimestampTz PgStartTime;
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index c83e4d5c20e..b4138a8f71d 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -72,8 +72,22 @@ $node->start;
# other tests are added to this file in the future
$node->safe_psql('postgres', "CREATE DATABASE test_log_connections");
+my $log_connections = $node->safe_psql('test_log_connections', q(SHOW log_connections;));
+is($log_connections, 'on', qq(check log connections has expected value 'on'));
+
+$node->connect_ok('test_log_connections',
+ qq(log_connections 'on' works as expected for backwards compatibility),
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],
+ log_unlike => [
+ qr/connection ready/,
+ ],);
+
$node->safe_psql('test_log_connections',
- q[ALTER SYSTEM SET log_connections = receipt,authorization;
+ q[ALTER SYSTEM SET log_connections = receipt,authorization,durations;
SELECT pg_reload_conf();]);
$node->connect_ok('test_log_connections',
@@ -81,6 +95,7 @@ $node->connect_ok('test_log_connections',
log_like => [
qr/connection received/,
qr/connection authorized: user=\S+ database=test_log_connections/,
+ qr/connection ready/,
],
log_unlike => [
qr/connection authenticated/,
@@ -95,6 +110,7 @@ $node->connect_ok('test_log_connections',
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized: user=\S+ database=test_log_connections/,
+ qr/connection ready/,
],);
# Authentication tests
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 29e52b64994..3da74999634 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
v13-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v13-0001-Modularize-log_connections-output.patchDownload
From 2496a51211395fc99162c6f4c2567b19a99aeb76 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:02:45 -0500
Subject: [PATCH v13 1/2] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC of the
connection aspects to log.
This gives users more control over the volume and kind of connection
logging.
The current log_connections options are 'receipt', 'authentication', and
'authorization'. The empty string disables all connection logging. 'all'
enables all available connection logging.
For backwards compatibility, the most common values for the
log_connections boolean are still supported (on, off, 1, 0, true,
false, yes, no).
This patch has much of the same behavior as [1] but was developed
independently.
[1] https://postgr.es/m/flat/CAA8Fd-qCB96uwfgMKrzfNs32mqqysi53yZFNVaRNJ6xDthZEgA%40mail.gmail.com
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 74 ++++++++-
src/backend/libpq/auth.c | 9 +-
src/backend/postmaster/postmaster.c | 1 -
src/backend/tcop/backend_startup.c | 154 +++++++++++++++++-
src/backend/utils/init/postinit.c | 3 +-
src/backend/utils/misc/guc_tables.c | 21 ++-
src/backend/utils/misc/postgresql.conf.sample | 4 +-
src/include/postmaster/postmaster.h | 1 -
src/include/tcop/backend_startup.h | 27 +++
src/include/utils/guc_hooks.h | 2 +
src/test/authentication/t/001_password.pl | 33 ++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 306 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..52f5a115391 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,84 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes aspects of each connection attempt to the server to
+ be logged. The default is the empty string, <literal>''</literal>,
+ which disables all connection logging. The following are the options
+ that may be specified:
+ </para>
+
+ <table id="log-connections-options">
+ <title>Log Connection Options</title>
+ <tgroup cols="2">
+ <colspec colname="col1" colwidth="1*"/>
+ <colspec colname="col2" colwidth="2*"/>
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>receipt</literal></entry>
+ <entry>Logs receipt of a connection.</entry>
+ </row>
+
+ <row>
+ <entry><literal>authentication</literal></entry>
+ <entry>
+ Logs the original identity used by an authentication method to
+ identify a user. In most cases, the identity string matches the
+ PostgreSQL username, but some third-party authentication methods
+ may alter the original user identifier before the server stores it.
+ Failed authentication is always logged regardless of the value of
+ this setting.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>authorization</literal></entry>
+ <entry>
+ Logs successful completion of authorization. At this point the
+ connection has been established but the backend is not yet fully
+ set up.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>all</literal></entry>
+ <entry>
+ A convenience alias equivalent to specifying all options. If
+ <literal>all</literal> is specified in a list of other options, all
+ connection aspects will be logged.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ For the purposes of backwards compatibility, <literal>on</literal>,
+ <literal>off</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>yes</literal>,
+ <literal>no</literal>, <literal>1</literal>, and <literal>0</literal>
+ are still supported. The positive values are equivalent to specifying
+ the <literal>received</literal>, <literal>authenticated</literal>, and
+ <literal>authorized</literal> options.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..d95679b4dc8 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -38,6 +38,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/ipc.h"
+#include "tcop/backend_startup.h"
#include "utils/memutils.h"
/*----------------------------------------------------------------
@@ -317,7 +318,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authenticated' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +351,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATION)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +635,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATION) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..b4f34c57a1b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,6 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
bool enable_bonjour = false;
char *bonjour_name;
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..fb0baeb4a24 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -34,13 +34,17 @@
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
+#include "utils/varlena.h"
/* GUCs */
bool Trace_connection_negotiation = false;
+uint32 log_connections = 0;
+char *log_connections_string = NULL;
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
@@ -48,6 +52,7 @@ static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void process_startup_packet_die(SIGNAL_ARGS);
static void StartupPacketTimeoutHandler(void);
+static bool validate_log_connections_options(List *elemlist, uint32 *flags);
/*
* Entry point for a new backend process.
@@ -201,8 +206,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIPT)
{
if (remote_port[0])
ereport(LOG,
@@ -924,3 +929,148 @@ StartupPacketTimeoutHandler(void)
{
_exit(1);
}
+
+/*
+ * Helper for the log_connections GUC check hook.
+ *
+ * `elemlist` is a listified version of the string input passed to the
+ * log_connections GUC check hook, check_log_connections().
+ * check_log_connections() is responsible for cleaning up `elemlist`.
+ *
+ * validate_log_connections_options() returns false if an error was
+ * encountered and the GUC input could not be validated and true otherwise.
+ *
+ * `flags` returns the flags that should be stored in the log_connections GUC
+ * by its assign hook.
+ */
+static bool
+validate_log_connections_options(List *elemlist, uint32 *flags)
+{
+ ListCell *l;
+ char *item;
+
+ /* For backwards compatibility, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
+
+ *flags = 0;
+
+ /* If an empty string was passed, we're done */
+ if (list_length(elemlist) == 0)
+ return true;
+
+ /*
+ * Now check for the backwards compatibility options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatibility option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
+ GUC_check_errdetail("Cannot specify log_connections option \"%s\" in a list with other options.",
+ item);
+ return false;
+ }
+
+ *flags = option.val;
+ return true;
+ }
+
+ /* Now check the aspect options. The empty string was already handled */
+ foreach(l, elemlist)
+ {
+ static const struct config_enum_entry options[] = {
+ {"receipt", LOG_CONNECTION_RECEIPT},
+ {"authentication", LOG_CONNECTION_AUTHENTICATION},
+ {"authorization", LOG_CONNECTION_AUTHORIZATION},
+ {"all", LOG_CONNECTION_ALL},
+ };
+
+ item = lfirst(l);
+ for (size_t i = 0; i < lengthof(options); i++)
+ {
+ struct config_enum_entry option = options[i];
+
+ if (pg_strcasecmp(item, option.name) == 0)
+ {
+ *flags |= option.val;
+ goto next;
+ }
+ }
+
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ return false;
+
+next: ;
+ }
+
+ return true;
+}
+
+
+/*
+ * GUC check hook for log_connections
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ uint32 flags;
+ char *rawstring;
+ List *elemlist;
+ bool success;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitIdentifierString(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Validation logic is all in the helper */
+ success = validate_log_connections_options(elemlist, &flags);
+
+ /* Time for cleanup */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ if (!success)
+ return false;
+
+ /*
+ * We succeeded, so allocate `extra` and save the flags there for use by
+ * assign_log_connections().
+ */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+}
+
+/*
+ * GUC assign hook for log_connections
+ */
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..6c207e17768 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -54,6 +54,7 @@
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -252,7 +253,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZATION)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..d95e6bb9e4b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4886,6 +4877,18 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs specified aspects of connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..aca610f4fad 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections=all". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,7 +578,7 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = ''
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..aa786bfacf3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,6 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..b5a33a2b0e1 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -16,6 +16,8 @@
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
+extern PGDLLIMPORT uint32 log_connections;
+extern PGDLLIMPORT char *log_connections_string;
/*
* CAC_state is passed from postmaster to the backend process, to indicate
@@ -39,6 +41,31 @@ typedef struct BackendStartupData
CAC_state canAcceptConnections;
} BackendStartupData;
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ *
+ * RECEIPT, AUTHENTICATION, and AUTHORIZATION are different aspects of
+ * connection establishment and backend setup for which we may emit a log
+ * message.
+ *
+ * ALL is a convenience alias equivalent to all of the above aspects.
+ *
+ * ON is backwards compatibility alias for the connection aspects that were
+ * logged in Postgres versions < 18.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIPT = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATION = (1 << 1),
+ LOG_CONNECTION_AUTHORIZATION = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIPT |
+ LOG_CONNECTION_AUTHENTICATION |
+ LOG_CONNECTION_AUTHORIZATION,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
+} LogConnectionOption;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 4ce22ccbdf2..c83e4d5c20e 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -66,6 +66,39 @@ $node->init;
$node->append_conf('postgresql.conf', "log_connections = on\n");
$node->start;
+# Test behavior of log_connections GUC
+
+# Make a database for the log_connections tests to avoid test fragility if
+# other tests are added to this file in the future
+$node->safe_psql('postgres', "CREATE DATABASE test_log_connections");
+
+$node->safe_psql('test_log_connections',
+ q[ALTER SYSTEM SET log_connections = receipt,authorization;
+ SELECT pg_reload_conf();]);
+
+$node->connect_ok('test_log_connections',
+ q(log_connections with subset of specified options logs only those aspects),
+ log_like => [
+ qr/connection received/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],
+ log_unlike => [
+ qr/connection authenticated/,
+ ],);
+
+$node->safe_psql('test_log_connections',
+ qq(ALTER SYSTEM SET log_connections = 'all'; SELECT pg_reload_conf();));
+
+$node->connect_ok('test_log_connections',
+ qq(log_connections 'all' logs all available connection aspects),
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],);
+
+# Authentication tests
+
# could fail in FIPS mode
my $md5_works = ($node->psql('postgres', "select md5('')") == 0);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9840060997f..29e52b64994 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1555,6 +1555,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
On Fri, Mar 7, 2025 at 10:09 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
Given that conn_timing.ready_for_use is only used here:
+ if (!reported_first_ready_for_query && + (log_connections & LOG_CONNECTION_READY_FOR_USE) && + IsConnectionBackend(MyBackendType)) + { + uint64 total_duration, + fork_duration, + auth_duration; + + conn_timing.ready_for_use = GetCurrentTimestamp(); + + total_duration = + TimestampDifferenceMicroseconds(conn_timing.socket_create, + conn_timing.ready_for_use);I wonder if ready_for_use needs to be part of ConnectionTiming after all.
I mean we could just:"
total_duration = TimestampDifferenceMicroseconds(conn_timing.socket_create, GetCurrentTimestamp())
"That said, having ready_for_use part of ConnectionTiming could be
useful if we decide to create a SQL API on top of this struct though. So,
that's probably fine and better as it is.
That's what I was thinking. It felt like good symmetry to have it there.
And if we keep it part of ConnectionTiming, then:
=== 02
+ if (!reported_first_ready_for_query && + (log_connections & LOG_CONNECTION_READY_FOR_USE) && + IsConnectionBackend(MyBackendType))What about getting rid of reported_first_ready_for_query and check if
conn_timing.ready_for_use is zero instead?
Unfortunately, I think a TimestampTz value of 0 corresponds to
1970-01-01 00:00:00 UTC, which is a valid timestamp.
- Melanie
On Fri, Mar 7, 2025 at 4:53 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
If I add a hypothetical auth method in the future that authenticates,
and then farms the authorization decision out to some slow-moving
network machinery, would "authenticated" retroactively become a stage
then? (OAuth almost does this today... but it's not quite separated
enough for me to claim it as an example.) If we call them "options"
instead, I guess we don't have to worry about shifting internals.
Yea, I could also see people wanting to add other messages in the
future that don't fit perfectly into the "stage" category. Keeping
things more general probably reduces that potential future friction
too.
In v13 that I shared here [1]/messages/by-id/CAAKRu_ZVecPK6QU-R_pFrkm6z4OAvE9PQBh1hL_GB8UW0cxTnw@mail.gmail.com I refer to them as "options" and, in
some contexts where that doesn't make sense, "aspects" of connection
establishment and backend setup. I also changed the tenses of the
other options from "received", "authenticated", and "authorized" to
"receipt", "authentication", "authorization". I think that makes them
sound less like stages and more like a type of message.
That also makes me wonder if the
"authenticated" log_connections option should actually be called
"auth_id" or something similar.To bikeshed the specific suggestion of "auth_id": both Peter E and
Robert have previously expressed concern that my internal name choice
of "authn_id" was too opaque, and that maybe I should have just
expanded the terms. Also, I think if one option is called
"authorized", the other half should probably be called "authenticated"
if for no other reason than symmetry. It also matches the prefix used
in the logs, for English builds.
Yea, "auth_id" is too ambiguous for a GUC option. I went with
"authorization" because it sounds a bit less like a stage while also
being a recognizable word. It achieves symmetry -- though it doesn't
match the prefix used in the logs.
- Melanie
[1]: /messages/by-id/CAAKRu_ZVecPK6QU-R_pFrkm6z4OAvE9PQBh1hL_GB8UW0cxTnw@mail.gmail.com
On Fri, Mar 7, 2025 at 2:17 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
Yea, "auth_id" is too ambiguous for a GUC option. I went with
"authorization" because it sounds a bit less like a stage while also
being a recognizable word. It achieves symmetry -- though it doesn't
match the prefix used in the logs.
Yeah, that sounds reasonable.
--Jacob
Hi,
On Fri, Mar 07, 2025 at 03:29:14PM -0500, Melanie Plageman wrote:
On Fri, Mar 7, 2025 at 6:16 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:But at the end, what we're "really" interested in this thread, given its $SUBJECT,
is to actually log the timings.I'm not sure that people would enable this option much if
it was called ready_for_use, since just logging when we are ready for
query is likely not that valuable.
Agreed.
So, I would call "received" and "authorized" stages and the
authentication ID message not a stage. I think I should not call these
"log_connections stages" in the docs and comments and instead call
them "log_connections options".
Makes sense to me.
And I am wondering if the "timings" option should be called "timings"
or "durations"? I want to convey that it is about printing connection
setup durations and not about whether or not we measure timings. But
log_connections=durations sounds more like it logs the total duration
of the connection setup and not component parts...
I agree that both could be misinterpreted (even if the documentation should
clear any doubts). Anyway, it's probably better to put more details in the
option name then, something like "setup_timings" maybe?
I actually think even if we change the option name, it could be
valuable to have the message begin with "connection ready for use";
otherwise, it's unclear when we are printing the message.
Good point, I do agree.
Logging
messages are very tied to what was happening when they were emitted.
Usually they include some context on when they were emitted. Thus, I
think it makes sense to somehow contextualize the message in this way
in the text. It does make the message rather long, though...
Just use "connection ready" maybe?
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On 7 Mar 2025, at 23:08, Melanie Plageman <melanieplageman@gmail.com> wrote:
Sorry for being late to the party. I like this functionality and would
definitely like to see it in v18.
An unrelated note about the attached v13:
I changed the language from log_connections "stages" to "options" and
"aspects" depending on the context. I also changed the name of the
option introduced in 0002 to "durations".
I like the use options/aspects here, it works better than stages IMO.
A few comments on the patch:
+ /* For backwards compatibility, we accept these tokens by themselves */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
It's not documented here that the backwards compatibility will turn the
LOG_CONNECTION_ON value into a set of the new fine-grained value. A brief
mention would be good for future code reading (or a hint to look for details in
src/include/tcop/backend_startup.h).
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIPT |
+ LOG_CONNECTION_AUTHENTICATION |
+ LOG_CONNECTION_AUTHORIZATION,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_ON,
Nitpick I know, but reusing the backwards compatible _ON for _ALL makes it seem
like they are the other way around (_ALL being for backwards compat). 0002
makes it less so but I would prefer to spend the extra lines of code and spell
out both.
+#define IsConnectionBackend(backend_type) \
This feels like a slightly too generic name to fully convey what it does. I
don't have a better one off the cuff but I had to refer back multiple times to
remind myself what it did.
+ backend at the time the connection is ready for use. The log
I'm not sure if "ready for use" will be clear for all readers? The other
options are using a bit more precise language.
+ {"durations", LOG_CONNECTION_DURATIONS},
I think "durations" is a good name for this, but we risk causing some confusion
in postgresql.conf when set, as it will end up like this:
log_connections = durations
#log_disconnections = off
#log_duration = off
How log_connections=durations and log_duration=off are related might not be
immediately obvious to new users. Can we avoid confusion by adding comments on
these log entries perhaps?
+ errmsg("connection ready: total=%.3f ms, fork=%.3f ms, authentication=%.3f ms",
The use of "total" here makes it seem like (fork+authentication)=total. Not
sure if we can do better and still be descriptive. Maybe it's a case of
documenting it if there are questions or complaints?
I'm a bit torn about having the tests in authentication/001_password.
Agreed, they're not a great fit and it's not where I would look for them. That
being said, paying for spinning up a new cluster for just this also doesn't
seem great. Maybe adding a comment in the test file explaining why they are in
there could help readers.
--
Daniel Gustafsson
On Mon, Mar 10, 2025 at 5:03 PM Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Mar 2025, at 23:08, Melanie Plageman <melanieplageman@gmail.com> wrote:
Sorry for being late to the party. I like this functionality and would
definitely like to see it in v18.
Thanks so much for the review!
+ /* For backwards compatibility, we accept these tokens by themselves */ + static const struct config_enum_entry compat_options[] = { + {"off", 0}, + {"false", 0}, + {"no", 0}, + {"0", 0}, + {"on", LOG_CONNECTION_ON}, + {"true", LOG_CONNECTION_ON}, + {"yes", LOG_CONNECTION_ON}, + {"1", LOG_CONNECTION_ON}, + }; It's not documented here that the backwards compatibility will turn the LOG_CONNECTION_ON value into a set of the new fine-grained value. A brief mention would be good for future code reading (or a hint to look for details in src/include/tcop/backend_startup.h).
I've added a comment about this in attached v14.
+ LOG_CONNECTION_ON = + LOG_CONNECTION_RECEIPT | + LOG_CONNECTION_AUTHENTICATION | + LOG_CONNECTION_AUTHORIZATION, + LOG_CONNECTION_ALL = + LOG_CONNECTION_ON, Nitpick I know, but reusing the backwards compatible _ON for _ALL makes it seem like they are the other way around (_ALL being for backwards compat). 0002 makes it less so but I would prefer to spend the extra lines of code and spell out both.
Done.
+#define IsConnectionBackend(backend_type) \
This feels like a slightly too generic name to fully convey what it does. I
don't have a better one off the cuff but I had to refer back multiple times to
remind myself what it did.
Is `IsClientBackend()` better? It doesn't sound immediately like it
includes WAL sender backends (replication connections), but those are
initiated by clients too, right?
I didn't change it in v14 because I wasn't sure it was better and
wanted to hear what you thought.
+ backend at the time the connection is ready for use. The log
I'm not sure if "ready for use" will be clear for all readers? The other
options are using a bit more precise language.
I've tried to clear this up.
+ {"durations", LOG_CONNECTION_DURATIONS},
I think "durations" is a good name for this, but we risk causing some confusion
in postgresql.conf when set, as it will end up like this:log_connections = durations
#log_disconnections = off
#log_duration = offHow log_connections=durations and log_duration=off are related might not be
immediately obvious to new users. Can we avoid confusion by adding comments on
these log entries perhaps?
I forgot about log_duration and that log_disconnections prints the
connection duration. Thinking about it more and taking into account
something Bertrand suggested upthread, I decided to rename the option
"setup_durations". It's a bit long, but it does convey that these
durations are only about connection setup.
I also added more details about what this is both to docs and
postgresql.conf.sample. Thanks!
+ errmsg("connection ready: total=%.3f ms, fork=%.3f ms, authentication=%.3f ms",
The use of "total" here makes it seem like (fork+authentication)=total. Not
sure if we can do better and still be descriptive. Maybe it's a case of
documenting it if there are questions or complaints?
I've changed it to "setup total=xxx ms, fork=xxx...". I also added
details about the component durations to the docs.
I'm a bit torn about having the tests in authentication/001_password.
Agreed, they're not a great fit and it's not where I would look for them. That
being said, paying for spinning up a new cluster for just this also doesn't
seem great. Maybe adding a comment in the test file explaining why they are in
there could help readers.
Yep, added the comment.
I did more manual testing of my patches, and I think they are mostly
ready for commit except for the IsConnectionBackend() macro (if we
have something to change it to).
I did notice one other thing, though. Because
process_startup_options() is called after all the existing
log_connections messages are emitted, setup_durations behaves a bit
differently than the other options. I actually think the other
log_connections options are not quite right, so perhaps it is okay
that the new option is different if it is more compliant.
For example, if you start from a freshly initdb'd cluster and do ALTER
SYSTEM SET log_connections='all', then restart or reload the conf file
and reconnect, you'll see the receipt, authentication, authorization,
and setup_durations messages. If you then `export PGOPTIONS="-c
log_connections="` and reconnect again, you'll see the receipt,
authentication, and authorization messages but not the setup_durations
message. The setup_durations message is emitted after we process
PGOPTIONS and set log_connections = ''. I think this behavior is okay,
though I'm not sure if/where it should be documented
- Melanie
Attachments:
v14-0001-Modularize-log_connections-output.patchtext/x-patch; charset=US-ASCII; name=v14-0001-Modularize-log_connections-output.patchDownload
From 46d416c95943ff79762c551443e6de1f64f771e0 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:02:45 -0500
Subject: [PATCH v14 1/2] Modularize log_connections output
Convert the boolean log_connections GUC into a list GUC comprised of the
connection aspects to log.
This gives users more control over the volume and kind of connection
logging.
The current log_connections options are 'receipt', 'authentication', and
'authorization'. The empty string disables all connection logging. 'all'
enables all available connection logging.
For backwards compatibility, the most common values for the
log_connections boolean are still supported (on, off, 1, 0, true,
false, yes, no).
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 77 ++++++++-
src/backend/libpq/auth.c | 9 +-
src/backend/postmaster/postmaster.c | 1 -
src/backend/tcop/backend_startup.c | 161 +++++++++++++++++-
src/backend/utils/init/postinit.c | 3 +-
src/backend/utils/misc/guc_tables.c | 21 ++-
src/backend/utils/misc/postgresql.conf.sample | 8 +-
src/include/postmaster/postmaster.h | 1 -
src/include/tcop/backend_startup.h | 29 ++++
src/include/utils/guc_hooks.h | 2 +
src/test/authentication/t/001_password.pl | 38 +++++
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 326 insertions(+), 25 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d2fa5f7d1a9..d939d99d6e8 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7315,20 +7315,87 @@ local0.* /var/log/postgresql
</varlistentry>
<varlistentry id="guc-log-connections" xreflabel="log_connections">
- <term><varname>log_connections</varname> (<type>boolean</type>)
+ <term><varname>log_connections</varname> (<type>string</type>)
<indexterm>
<primary><varname>log_connections</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
- Causes each attempted connection to the server to be logged,
- as well as successful completion of both client authentication (if
- necessary) and authorization.
+ Causes aspects of each connection attempt to the server to be
+ logged. The default is the empty string, <literal>''</literal>,
+ which disables all connection logging. The following options may
+ be specified alone or in a comma-separated list:
+ </para>
+
+ <table id="log-connections-options">
+ <title>Log Connection Options</title>
+ <tgroup cols="2">
+ <colspec colname="col1" colwidth="1*"/>
+ <colspec colname="col2" colwidth="2*"/>
+ <thead>
+ <row>
+ <entry>Name</entry>
+ <entry>Description</entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><literal>receipt</literal></entry>
+ <entry>Logs receipt of a connection.</entry>
+ </row>
+
+ <row>
+ <entry><literal>authentication</literal></entry>
+ <entry>
+ Logs the original identity used by an authentication method
+ to identify a user. In most cases, the identity string
+ matches the PostgreSQL username, but some third-party
+ authentication methods may alter the original user
+ identifier before the server stores it. Failed
+ authentication is always logged regardless of the value of
+ this setting.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>authorization</literal></entry>
+ <entry>
+ Logs successful completion of authorization. At this point
+ the connection has been established but the backend is not
+ yet fully set up. The log message includes the authorized
+ username as well as the database name and application name,
+ if applicable.
+ </entry>
+ </row>
+
+ <row>
+ <entry><literal>all</literal></entry>
+ <entry>
+ A convenience alias equivalent to specifying all options. If
+ <literal>all</literal> is specified in a list of other
+ options, all connection aspects will be logged.
+ </entry>
+ </row>
+
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ For the purposes of backwards compatibility, <literal>on</literal>,
+ <literal>off</literal>, <literal>true</literal>,
+ <literal>false</literal>, <literal>yes</literal>,
+ <literal>no</literal>, <literal>1</literal>, and <literal>0</literal>
+ are still supported. The positive values are equivalent to specifying
+ the <literal>receipt</literal>, <literal>authentication</literal>, and
+ <literal>authorization</literal> options.
+ </para>
+
+ <para>
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this parameter at session start,
and it cannot be changed at all within a session.
- The default is <literal>off</literal>.
</para>
<note>
diff --git a/src/backend/libpq/auth.c b/src/backend/libpq/auth.c
index 81e2f8184e3..e18683c47e7 100644
--- a/src/backend/libpq/auth.c
+++ b/src/backend/libpq/auth.c
@@ -38,6 +38,7 @@
#include "postmaster/postmaster.h"
#include "replication/walsender.h"
#include "storage/ipc.h"
+#include "tcop/backend_startup.h"
#include "utils/memutils.h"
/*----------------------------------------------------------------
@@ -317,7 +318,8 @@ auth_failed(Port *port, int status, const char *logdetail)
/*
* Sets the authenticated identity for the current user. The provided string
* will be stored into MyClientConnectionInfo, alongside the current HBA
- * method in use. The ID will be logged if log_connections is enabled.
+ * method in use. The ID will be logged if log_connections has the
+ * 'authentication' option specified.
*
* Auth methods should call this routine exactly once, as soon as the user is
* successfully authenticated, even if they have reasons to know that
@@ -349,7 +351,7 @@ set_authn_id(Port *port, const char *id)
MyClientConnectionInfo.authn_id = MemoryContextStrdup(TopMemoryContext, id);
MyClientConnectionInfo.auth_method = port->hba->auth_method;
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHENTICATION)
{
ereport(LOG,
errmsg("connection authenticated: identity=\"%s\" method=%s "
@@ -633,7 +635,8 @@ ClientAuthentication(Port *port)
#endif
}
- if (Log_connections && status == STATUS_OK &&
+ if ((log_connections & LOG_CONNECTION_AUTHENTICATION) &&
+ status == STATUS_OK &&
!MyClientConnectionInfo.authn_id)
{
/*
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index d2a7a7add6f..b4f34c57a1b 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -237,7 +237,6 @@ int PreAuthDelay = 0;
int AuthenticationTimeout = 60;
bool log_hostname; /* for ps display and logging */
-bool Log_connections = false;
bool enable_bonjour = false;
char *bonjour_name;
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index c70746fa562..962b6e60002 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -34,13 +34,17 @@
#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
+#include "utils/guc_hooks.h"
#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
#include "utils/timeout.h"
+#include "utils/varlena.h"
/* GUCs */
bool Trace_connection_negotiation = false;
+uint32 log_connections = 0;
+char *log_connections_string = NULL;
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
@@ -48,6 +52,7 @@ static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void process_startup_packet_die(SIGNAL_ARGS);
static void StartupPacketTimeoutHandler(void);
+static bool validate_log_connections_options(List *elemlist, uint32 *flags);
/*
* Entry point for a new backend process.
@@ -201,8 +206,8 @@ BackendInitialize(ClientSocket *client_sock, CAC_state cac)
port->remote_host = MemoryContextStrdup(TopMemoryContext, remote_host);
port->remote_port = MemoryContextStrdup(TopMemoryContext, remote_port);
- /* And now we can issue the Log_connections message, if wanted */
- if (Log_connections)
+ /* And now we can log that the connection was received, if enabled */
+ if (log_connections & LOG_CONNECTION_RECEIPT)
{
if (remote_port[0])
ereport(LOG,
@@ -924,3 +929,155 @@ StartupPacketTimeoutHandler(void)
{
_exit(1);
}
+
+/*
+ * Helper for the log_connections GUC check hook.
+ *
+ * `elemlist` is a listified version of the string input passed to the
+ * log_connections GUC check hook, check_log_connections().
+ * check_log_connections() is responsible for cleaning up `elemlist`.
+ *
+ * validate_log_connections_options() returns false if an error was
+ * encountered and the GUC input could not be validated and true otherwise.
+ *
+ * `flags` returns the flags that should be stored in the log_connections GUC
+ * by its assign hook.
+ */
+static bool
+validate_log_connections_options(List *elemlist, uint32 *flags)
+{
+ ListCell *l;
+ char *item;
+
+ /*
+ * For backwards compatibility, we accept these tokens by themselves.
+ *
+ * Prior to PostgreSQL 18, log_connections was a boolean GUC that accepted
+ * any unambiguous substring of 'true', 'false', 'yes', 'no', 'on', and
+ * 'off'. Since log_connections became a list of strings in 18, we only
+ * accept complete option strings.
+ */
+ static const struct config_enum_entry compat_options[] = {
+ {"off", 0},
+ {"false", 0},
+ {"no", 0},
+ {"0", 0},
+ {"on", LOG_CONNECTION_ON},
+ {"true", LOG_CONNECTION_ON},
+ {"yes", LOG_CONNECTION_ON},
+ {"1", LOG_CONNECTION_ON},
+ };
+
+ *flags = 0;
+
+ /* If an empty string was passed, we're done */
+ if (list_length(elemlist) == 0)
+ return true;
+
+ /*
+ * Now check for the backwards compatibility options. They must always be
+ * specified on their own, so we error out if the first option is a
+ * backwards compatibility option and other options are also specified.
+ */
+ item = linitial(elemlist);
+
+ for (size_t i = 0; i < lengthof(compat_options); i++)
+ {
+ struct config_enum_entry option = compat_options[i];
+
+ if (pg_strcasecmp(item, option.name) != 0)
+ continue;
+
+ if (list_length(elemlist) > 1)
+ {
+ GUC_check_errdetail("Cannot specify log_connections option \"%s\" in a list with other options.",
+ item);
+ return false;
+ }
+
+ *flags = option.val;
+ return true;
+ }
+
+ /* Now check the aspect options. The empty string was already handled */
+ foreach(l, elemlist)
+ {
+ static const struct config_enum_entry options[] = {
+ {"receipt", LOG_CONNECTION_RECEIPT},
+ {"authentication", LOG_CONNECTION_AUTHENTICATION},
+ {"authorization", LOG_CONNECTION_AUTHORIZATION},
+ {"all", LOG_CONNECTION_ALL},
+ };
+
+ item = lfirst(l);
+ for (size_t i = 0; i < lengthof(options); i++)
+ {
+ struct config_enum_entry option = options[i];
+
+ if (pg_strcasecmp(item, option.name) == 0)
+ {
+ *flags |= option.val;
+ goto next;
+ }
+ }
+
+ GUC_check_errdetail("Invalid option \"%s\".", item);
+ return false;
+
+next: ;
+ }
+
+ return true;
+}
+
+
+/*
+ * GUC check hook for log_connections
+ */
+bool
+check_log_connections(char **newval, void **extra, GucSource source)
+{
+ uint32 flags;
+ char *rawstring;
+ List *elemlist;
+ bool success;
+
+ /* Need a modifiable copy of string */
+ rawstring = pstrdup(*newval);
+
+ if (!SplitIdentifierString(rawstring, ',', &elemlist))
+ {
+ GUC_check_errdetail("Invalid list syntax in parameter \"log_connections\".");
+ pfree(rawstring);
+ list_free(elemlist);
+ return false;
+ }
+
+ /* Validation logic is all in the helper */
+ success = validate_log_connections_options(elemlist, &flags);
+
+ /* Time for cleanup */
+ pfree(rawstring);
+ list_free(elemlist);
+
+ if (!success)
+ return false;
+
+ /*
+ * We succeeded, so allocate `extra` and save the flags there for use by
+ * assign_log_connections().
+ */
+ *extra = guc_malloc(ERROR, sizeof(int));
+ *((int *) *extra) = flags;
+
+ return true;
+}
+
+/*
+ * GUC assign hook for log_connections
+ */
+void
+assign_log_connections(const char *newval, void *extra)
+{
+ log_connections = *((int *) extra);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index ee1a9d5d98b..6c207e17768 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -54,6 +54,7 @@
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
#include "storage/sync.h"
+#include "tcop/backend_startup.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/builtins.h"
@@ -252,7 +253,7 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
- if (Log_connections)
+ if (log_connections & LOG_CONNECTION_AUTHORIZATION)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index ad25cbb39c5..d95e6bb9e4b 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1219,15 +1219,6 @@ struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
- {
- {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
- gettext_noop("Logs each successful connection."),
- NULL
- },
- &Log_connections,
- false,
- NULL, NULL, NULL
- },
{
{"trace_connection_negotiation", PGC_POSTMASTER, DEVELOPER_OPTIONS,
gettext_noop("Logs details of pre-authentication connection handshake."),
@@ -4886,6 +4877,18 @@ struct config_string ConfigureNamesString[] =
NULL, NULL, NULL
},
+ {
+ {"log_connections", PGC_SU_BACKEND, LOGGING_WHAT,
+ gettext_noop("Logs specified aspects of connection establishment."),
+ NULL,
+ GUC_LIST_INPUT
+ },
+ &log_connections_string,
+ "",
+ check_log_connections, assign_log_connections, NULL
+ },
+
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, NULL, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2d1de9c37bd..c291c05d181 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -21,7 +21,7 @@
# require a server shutdown and restart to take effect.
#
# Any parameter can also be given as a command-line option to the server, e.g.,
-# "postgres -c log_connections=on". Some parameters can be changed at run time
+# "postgres -c log_connections=all". Some parameters can be changed at run time
# with the "SET" SQL command.
#
# Memory units: B = bytes Time units: us = microseconds
@@ -578,9 +578,11 @@
# actions running at least this number
# of milliseconds.
#log_checkpoints = on
-#log_connections = off
+#log_connections = '' # log aspects of connection setup
+ # options include receipt, authentication, authorization,
+ # and all to log all of these aspects
#log_disconnections = off
-#log_duration = off
+#log_duration = off # log statement duration
#log_error_verbosity = default # terse, default, or verbose messages
#log_hostname = off
#log_line_prefix = '%m [%p] ' # special values:
diff --git a/src/include/postmaster/postmaster.h b/src/include/postmaster/postmaster.h
index b6a3f275a1b..aa786bfacf3 100644
--- a/src/include/postmaster/postmaster.h
+++ b/src/include/postmaster/postmaster.h
@@ -63,7 +63,6 @@ extern PGDLLIMPORT char *ListenAddresses;
extern PGDLLIMPORT bool ClientAuthInProgress;
extern PGDLLIMPORT int PreAuthDelay;
extern PGDLLIMPORT int AuthenticationTimeout;
-extern PGDLLIMPORT bool Log_connections;
extern PGDLLIMPORT bool log_hostname;
extern PGDLLIMPORT bool enable_bonjour;
extern PGDLLIMPORT char *bonjour_name;
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index 73285611203..e00851a004e 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -16,6 +16,8 @@
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
+extern PGDLLIMPORT uint32 log_connections;
+extern PGDLLIMPORT char *log_connections_string;
/*
* CAC_state is passed from postmaster to the backend process, to indicate
@@ -39,6 +41,33 @@ typedef struct BackendStartupData
CAC_state canAcceptConnections;
} BackendStartupData;
+/*
+ * Granular control over which messages to log for the log_connections GUC.
+ *
+ * RECEIPT, AUTHENTICATION, and AUTHORIZATION are different aspects of
+ * connection establishment and backend setup for which we may emit a log
+ * message.
+ *
+ * ALL is a convenience alias equivalent to all of the above aspects.
+ *
+ * ON is backwards compatibility alias for the connection aspects that were
+ * logged in Postgres versions < 18.
+ */
+typedef enum LogConnectionOption
+{
+ LOG_CONNECTION_RECEIPT = (1 << 0),
+ LOG_CONNECTION_AUTHENTICATION = (1 << 1),
+ LOG_CONNECTION_AUTHORIZATION = (1 << 2),
+ LOG_CONNECTION_ON =
+ LOG_CONNECTION_RECEIPT |
+ LOG_CONNECTION_AUTHENTICATION |
+ LOG_CONNECTION_AUTHORIZATION,
+ LOG_CONNECTION_ALL =
+ LOG_CONNECTION_RECEIPT |
+ LOG_CONNECTION_AUTHENTICATION |
+ LOG_CONNECTION_AUTHORIZATION,
+} LogConnectionOption;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 951451a9765..9a0d8ec85c7 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -51,6 +51,8 @@ extern bool check_datestyle(char **newval, void **extra, GucSource source);
extern void assign_datestyle(const char *newval, void *extra);
extern bool check_debug_io_direct(char **newval, void **extra, GucSource source);
extern void assign_debug_io_direct(const char *newval, void *extra);
+extern bool check_log_connections(char **newval, void **extra, GucSource source);
+extern void assign_log_connections(const char *newval, void *extra);
extern bool check_default_table_access_method(char **newval, void **extra,
GucSource source);
extern bool check_default_tablespace(char **newval, void **extra,
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index 4ce22ccbdf2..e307dee5c48 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -7,6 +7,8 @@
# - MD5-encrypted
# - SCRAM-encrypted
# This test can only run with Unix-domain sockets.
+#
+# There's also a few tests of the log_connections GUC here.
use strict;
use warnings FATAL => 'all';
@@ -66,6 +68,42 @@ $node->init;
$node->append_conf('postgresql.conf', "log_connections = on\n");
$node->start;
+# Test behavior of log_connections GUC
+#
+# There wasn't another test file where these tests obviously fit, and we don't
+# want to incur the cost of spinning up a new cluster just to test one GUC.
+
+# Make a database for the log_connections tests to avoid test fragility if
+# other tests are added to this file in the future
+$node->safe_psql('postgres', "CREATE DATABASE test_log_connections");
+
+$node->safe_psql('test_log_connections',
+ q[ALTER SYSTEM SET log_connections = receipt,authorization;
+ SELECT pg_reload_conf();]);
+
+$node->connect_ok('test_log_connections',
+ q(log_connections with subset of specified options logs only those aspects),
+ log_like => [
+ qr/connection received/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],
+ log_unlike => [
+ qr/connection authenticated/,
+ ],);
+
+$node->safe_psql('test_log_connections',
+ qq(ALTER SYSTEM SET log_connections = 'all'; SELECT pg_reload_conf();));
+
+$node->connect_ok('test_log_connections',
+ qq(log_connections 'all' logs all available connection aspects),
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],);
+
+# Authentication tests
+
# could fail in FIPS mode
my $md5_works = ($node->psql('postgres', "select md5('')") == 0);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dfe2690bdd3..9e7f58adb18 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1556,6 +1556,7 @@ LockTupleMode
LockViewRecurse_context
LockWaitPolicy
LockingClause
+LogConnectionOption
LogOpts
LogStmtLevel
LogicalDecodeBeginCB
--
2.34.1
v14-0002-Add-connection-establishment-duration-logging.patchtext/x-patch; charset=US-ASCII; name=v14-0002-Add-connection-establishment-duration-logging.patchDownload
From f307fb571230e8aa452a19d68a25933f500df4f4 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 5 Mar 2025 18:05:19 -0500
Subject: [PATCH v14 2/2] Add connection establishment duration logging
Add log_connections option 'setup_durations' which logs durations of
several key parts of connection establishment and backend setup.
For an incoming connection, starting from when the postmaster gets a
socket from accept() and ending when the forked child backend is first
ready for query, there are multiple steps that could each take longer
than expected due to external factors. This logging provides visibility
into authentication and fork duration as well as the end-to-end
connection establishment and backend initialization time.
To make this portable, the timings captured in the postmaster (socket
creation time, fork initiation time) are passed through the
BackendStartupData.
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by: Fujii Masao <masao.fujii@oss.nttdata.com>
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Jelte Fennema-Nio <postgres@jeltef.nl>
Reviewed-by: Guillaume Lelarge <guillaume.lelarge@dalibo.com>
Discussion: https://postgr.es/m/flat/CAAKRu_b_smAHK0ZjrnL5GRxnAVWujEXQWpLXYzGbmpcZd3nLYw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 13 +++++
src/backend/postmaster/launch_backend.c | 31 ++++++++++
src/backend/postmaster/postmaster.c | 6 ++
src/backend/tcop/backend_startup.c | 11 ++++
src/backend/tcop/postgres.c | 33 +++++++++++
src/backend/utils/init/postinit.c | 6 ++
src/backend/utils/misc/postgresql.conf.sample | 2 +-
src/include/miscadmin.h | 7 +++
src/include/tcop/backend_startup.h | 57 +++++++++++++++++--
src/include/utils/timestamp.h | 9 +++
src/test/authentication/t/001_password.pl | 18 +++++-
src/tools/pgindent/typedefs.list | 1 +
12 files changed, 188 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d939d99d6e8..403d959ab30 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7369,6 +7369,19 @@ local0.* /var/log/postgresql
</entry>
</row>
+ <row>
+ <entry><literal>setup_durations</literal></entry>
+ <entry>
+ Logs the time spent establishing the connection and setting up the
+ backend at the time the connection is ready to execute its first
+ query. The log message includes the total setup duration, starting
+ from the postmaster accepting the incoming connection and ending
+ when the connection is ready for query. It also includes the time
+ it took to fork the new backend and the time it took to
+ authenticate the user.
+ </entry>
+ </row>
+
<row>
<entry><literal>all</literal></entry>
<entry>
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 47375e5bfaa..74b0c33adaa 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -232,6 +232,10 @@ postmaster_child_launch(BackendType child_type, int child_slot,
Assert(IsPostmasterEnvironment && !IsUnderPostmaster);
+ /* Capture time Postmaster initiates process creation for logging */
+ if (IsConnectionBackend(child_type))
+ ((BackendStartupData *) startup_data)->fork_started = GetCurrentTimestamp();
+
#ifdef EXEC_BACKEND
pid = internal_forkexec(child_process_kinds[child_type].name, child_slot,
startup_data, startup_data_len, client_sock);
@@ -240,6 +244,16 @@ postmaster_child_launch(BackendType child_type, int child_slot,
pid = fork_process();
if (pid == 0) /* child */
{
+ /* Capture and transfer timings that may be needed for logging */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = GetCurrentTimestamp();
+ }
+
/* Close the postmaster's sockets */
ClosePostmasterPorts(child_type == B_LOGGER);
@@ -586,11 +600,18 @@ SubPostmasterMain(int argc, char *argv[])
char *child_kind;
BackendType child_type;
bool found = false;
+ TimestampTz fork_end;
/* In EXEC_BACKEND case we will not have inherited these settings */
IsPostmasterEnvironment = true;
whereToSendOutput = DestNone;
+ /*
+ * Capture the end of process creation for logging. We don't include the
+ * time spent copying data from shared memory and setting up the backend.
+ */
+ fork_end = GetCurrentTimestamp();
+
/* Setup essential subsystems (to ensure elog() behaves sanely) */
InitializeGUCOptions();
@@ -648,6 +669,16 @@ SubPostmasterMain(int argc, char *argv[])
/* Read in remaining GUC variables */
read_nondefault_variables();
+ /* Capture and transfer timings that may be needed for log_connections */
+ if (IsConnectionBackend(child_type))
+ {
+ conn_timing.socket_create =
+ ((BackendStartupData *) startup_data)->socket_created;
+ conn_timing.fork_start =
+ ((BackendStartupData *) startup_data)->fork_started;
+ conn_timing.fork_end = fork_end;
+ }
+
/*
* Check that the data directory looks valid, which will also check the
* privileges on the data directory and update our umask and file/group
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index b4f34c57a1b..57155c00e01 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -3477,6 +3477,12 @@ BackendStartup(ClientSocket *client_sock)
BackendStartupData startup_data;
CAC_state cac;
+ /*
+ * Capture time that Postmaster got a socket from accept (for logging
+ * connection establishment and setup total duration).
+ */
+ startup_data.socket_created = GetCurrentTimestamp();
+
/*
* Allocate and assign the child slot. Note we must do this before
* forking, so that we can handle failures (out of memory or child-process
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index 962b6e60002..27c0b3c2b04 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -46,6 +46,16 @@ bool Trace_connection_negotiation = false;
uint32 log_connections = 0;
char *log_connections_string = NULL;
+/* Other globals */
+
+/*
+ * ConnectionTiming stores timestamps of various points in connection
+ * establishment and setup.
+ * ready_for_use is initialized to a special value here so we can check if
+ * we've already set it before doing so in PostgresMain().
+ */
+ConnectionTiming conn_timing = {.ready_for_use = TIMESTAMP_MINUS_INFINITY};
+
static void BackendInitialize(ClientSocket *client_sock, CAC_state cac);
static int ProcessSSLStartup(Port *port);
static int ProcessStartupPacket(Port *port, bool ssl_done, bool gss_done);
@@ -1006,6 +1016,7 @@ validate_log_connections_options(List *elemlist, uint32 *flags)
{"receipt", LOG_CONNECTION_RECEIPT},
{"authentication", LOG_CONNECTION_AUTHENTICATION},
{"authorization", LOG_CONNECTION_AUTHORIZATION},
+ {"setup_durations", LOG_CONNECTION_SETUP_DURATIONS},
{"all", LOG_CONNECTION_ALL},
};
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 947ffb40421..f2cae40b53d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -66,6 +66,7 @@
#include "storage/proc.h"
#include "storage/procsignal.h"
#include "storage/sinval.h"
+#include "tcop/backend_startup.h"
#include "tcop/fastpath.h"
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
@@ -4607,6 +4608,38 @@ PostgresMain(const char *dbname, const char *username)
/* Report any recently-changed GUC options */
ReportChangedGUCOptions();
+ /*
+ * The first time this backend is ready for query, log the
+ * durations of the different components of connection
+ * establishment and setup.
+ */
+ if (conn_timing.ready_for_use == TIMESTAMP_MINUS_INFINITY &&
+ (log_connections & LOG_CONNECTION_SETUP_DURATIONS) &&
+ IsConnectionBackend(MyBackendType))
+ {
+ uint64 total_duration,
+ fork_duration,
+ auth_duration;
+
+ conn_timing.ready_for_use = GetCurrentTimestamp();
+
+ total_duration =
+ TimestampDifferenceMicroseconds(conn_timing.socket_create,
+ conn_timing.ready_for_use);
+ fork_duration =
+ TimestampDifferenceMicroseconds(conn_timing.fork_start,
+ conn_timing.fork_end);
+ auth_duration =
+ TimestampDifferenceMicroseconds(conn_timing.auth_start,
+ conn_timing.auth_end);
+
+ ereport(LOG,
+ errmsg("connection ready: setup total=%.3f ms, fork=%.3f ms, authentication=%.3f ms",
+ (double) total_duration / NS_PER_US,
+ (double) fork_duration / NS_PER_US,
+ (double) auth_duration / NS_PER_US));
+ }
+
ReadyForQuery(whereToSendOutput);
send_ready_for_query = false;
}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 6c207e17768..4b2faf1ba9d 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -235,6 +235,9 @@ PerformAuthentication(Port *port)
}
#endif
+ /* Capture authentication start time for logging */
+ conn_timing.auth_start = GetCurrentTimestamp();
+
/*
* Set up a timeout in case a buggy or malicious client fails to respond
* during authentication. Since we're inside a transaction and might do
@@ -253,6 +256,9 @@ PerformAuthentication(Port *port)
*/
disable_timeout(STATEMENT_TIMEOUT, false);
+ /* Capture authentication end time for logging */
+ conn_timing.auth_end = GetCurrentTimestamp();
+
if (log_connections & LOG_CONNECTION_AUTHORIZATION)
{
StringInfoData logmsg;
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c291c05d181..d2bd329a587 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -580,7 +580,7 @@
#log_checkpoints = on
#log_connections = '' # log aspects of connection setup
# options include receipt, authentication, authorization,
- # and all to log all of these aspects
+ # setup_durations, and all to log all of these aspects
#log_disconnections = off
#log_duration = off # log statement duration
#log_error_verbosity = default # terse, default, or verbose messages
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index a2b63495eec..312eb3f9529 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -394,6 +394,13 @@ extern PGDLLIMPORT BackendType MyBackendType;
(AmAutoVacuumLauncherProcess() || \
AmLogicalSlotSyncWorkerProcess())
+/*
+ * Backend types that are spawned by the postmaster to serve a client
+ * or replication connection.
+ */
+#define IsConnectionBackend(backend_type) \
+ (backend_type == B_BACKEND || backend_type == B_WAL_SENDER)
+
extern const char *GetBackendTypeDesc(BackendType backendType);
extern void SetDatabasePath(const char *path);
diff --git a/src/include/tcop/backend_startup.h b/src/include/tcop/backend_startup.h
index e00851a004e..2912ef80288 100644
--- a/src/include/tcop/backend_startup.h
+++ b/src/include/tcop/backend_startup.h
@@ -14,11 +14,16 @@
#ifndef BACKEND_STARTUP_H
#define BACKEND_STARTUP_H
+#include "utils/timestamp.h"
+
/* GUCs */
extern PGDLLIMPORT bool Trace_connection_negotiation;
extern PGDLLIMPORT uint32 log_connections;
extern PGDLLIMPORT char *log_connections_string;
+/* Other globals */
+extern PGDLLIMPORT struct ConnectionTiming conn_timing;
+
/*
* CAC_state is passed from postmaster to the backend process, to indicate
* whether the connection should be accepted, or if the process should just
@@ -39,14 +44,27 @@ typedef enum CAC_state
typedef struct BackendStartupData
{
CAC_state canAcceptConnections;
+
+ /*
+ * Time at which the connection client socket is created. Only used for
+ * client and wal sender connections.
+ */
+ TimestampTz socket_created;
+
+ /*
+ * Time at which the postmaster initiates process creation -- either
+ * through fork or otherwise. Only used for client and wal sender
+ * connections.
+ */
+ TimestampTz fork_started;
} BackendStartupData;
/*
* Granular control over which messages to log for the log_connections GUC.
*
- * RECEIPT, AUTHENTICATION, and AUTHORIZATION are different aspects of
- * connection establishment and backend setup for which we may emit a log
- * message.
+ * RECEIPT, AUTHENTICATION, AUTHORIZATION, and SETUP_DURATIONS are different
+ * aspects of connection establishment and backend setup for which we may emit
+ * a log message.
*
* ALL is a convenience alias equivalent to all of the above aspects.
*
@@ -58,6 +76,7 @@ typedef enum LogConnectionOption
LOG_CONNECTION_RECEIPT = (1 << 0),
LOG_CONNECTION_AUTHENTICATION = (1 << 1),
LOG_CONNECTION_AUTHORIZATION = (1 << 2),
+ LOG_CONNECTION_SETUP_DURATIONS = (1 << 3),
LOG_CONNECTION_ON =
LOG_CONNECTION_RECEIPT |
LOG_CONNECTION_AUTHENTICATION |
@@ -65,9 +84,39 @@ typedef enum LogConnectionOption
LOG_CONNECTION_ALL =
LOG_CONNECTION_RECEIPT |
LOG_CONNECTION_AUTHENTICATION |
- LOG_CONNECTION_AUTHORIZATION,
+ LOG_CONNECTION_AUTHORIZATION |
+ LOG_CONNECTION_SETUP_DURATIONS,
} LogConnectionOption;
+/*
+ * A collection of timings of various stages of connection establishment and
+ * setup for client backends and WAL senders.
+ *
+ * Used to emit the setup_durations log message for the log_connections GUC.
+ */
+typedef struct ConnectionTiming
+{
+ /*
+ * The time at which the client socket is created and the time at which
+ * the connection is fully set up and first ready for query. Together
+ * these represent the total connection establishment and setup time.
+ */
+ TimestampTz socket_create;
+ TimestampTz ready_for_use;
+
+ /* Time at which process creation was initiated */
+ TimestampTz fork_start;
+
+ /* Time at which process creation was completed */
+ TimestampTz fork_end;
+
+ /* Time at which authentication started */
+ TimestampTz auth_start;
+
+ /* Time at which authentication was finished */
+ TimestampTz auth_end;
+} ConnectionTiming;
+
extern void BackendMain(const void *startup_data, size_t startup_data_len) pg_attribute_noreturn();
#endif /* BACKEND_STARTUP_H */
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index 9963bddc0ec..8c205859c3b 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -85,6 +85,15 @@ IntervalPGetDatum(const Interval *X)
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+/* Helper for simple subtraction between two timestamps */
+static inline uint64
+TimestampDifferenceMicroseconds(TimestampTz start_time,
+ TimestampTz stop_time)
+{
+ if (start_time >= stop_time)
+ return 0;
+ return (uint64) stop_time - start_time;
+}
/* Set at postmaster start */
extern PGDLLIMPORT TimestampTz PgStartTime;
diff --git a/src/test/authentication/t/001_password.pl b/src/test/authentication/t/001_password.pl
index e307dee5c48..8269c470b59 100644
--- a/src/test/authentication/t/001_password.pl
+++ b/src/test/authentication/t/001_password.pl
@@ -77,8 +77,22 @@ $node->start;
# other tests are added to this file in the future
$node->safe_psql('postgres', "CREATE DATABASE test_log_connections");
+my $log_connections = $node->safe_psql('test_log_connections', q(SHOW log_connections;));
+is($log_connections, 'on', qq(check log connections has expected value 'on'));
+
+$node->connect_ok('test_log_connections',
+ qq(log_connections 'on' works as expected for backwards compatibility),
+ log_like => [
+ qr/connection received/,
+ qr/connection authenticated/,
+ qr/connection authorized: user=\S+ database=test_log_connections/,
+ ],
+ log_unlike => [
+ qr/connection ready/,
+ ],);
+
$node->safe_psql('test_log_connections',
- q[ALTER SYSTEM SET log_connections = receipt,authorization;
+ q[ALTER SYSTEM SET log_connections = receipt,authorization,setup_durations;
SELECT pg_reload_conf();]);
$node->connect_ok('test_log_connections',
@@ -86,6 +100,7 @@ $node->connect_ok('test_log_connections',
log_like => [
qr/connection received/,
qr/connection authorized: user=\S+ database=test_log_connections/,
+ qr/connection ready/,
],
log_unlike => [
qr/connection authenticated/,
@@ -100,6 +115,7 @@ $node->connect_ok('test_log_connections',
qr/connection received/,
qr/connection authenticated/,
qr/connection authorized: user=\S+ database=test_log_connections/,
+ qr/connection ready/,
],);
# Authentication tests
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9e7f58adb18..a2e592dbbbb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -484,6 +484,7 @@ ConnParams
ConnStatusType
ConnType
ConnectionStateEnum
+ConnectionTiming
ConsiderSplitContext
Const
ConstrCheck
--
2.34.1
On Tue, Mar 11, 2025 at 6:27 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
I did more manual testing of my patches, and I think they are mostly
ready for commit except for the IsConnectionBackend() macro (if we
have something to change it to).
I've committed this and marked it as such in the CF app.
Thanks to everyone for the review.
- Melanie
On 12.03.25 16:43, Melanie Plageman wrote:
On Tue, Mar 11, 2025 at 6:27 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:I did more manual testing of my patches, and I think they are mostly
ready for commit except for the IsConnectionBackend() macro (if we
have something to change it to).I've committed this and marked it as such in the CF app.
Thanks to everyone for the review.
log_connections has been changed from a Boolean parameter to a string
one, but a number of places in the documentation and in various pieces
of test code still use the old values. I think it would be better if
they were adjusted to the new style.
There are two places in doc/src/sgml/config.sgml where
log_connections=yes is used as an example. This is a relatively
prominent place, so it should not use deprecated values.
In src/backend/tcop/postgres.c, there is a call
SetConfigOption("log_connections", "true", context, source);
that could be adjusted.
Various uses in test code:
src/interfaces/libpq/t/005_negotiate_encryption.pl
src/test/authentication/t/001_password.pl
src/test/authentication/t/003_peer.pl
src/test/authentication/t/005_sspi.pl
src/test/authentication/t/007_pre_auth.pl
src/test/kerberos/t/001_auth.pl
src/test/ldap/t/001_auth.pl
src/test/ldap/t/002_bindpasswd.pl
src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
src/test/modules/oauth_validator/t/001_server.pl
src/test/modules/oauth_validator/t/002_client.pl
src/test/postmaster/t/002_connection_limits.pl
src/test/postmaster/t/003_start_stop.pl
src/test/recovery/t/013_crash_restart.pl
src/test/recovery/t/022_crash_temp_files.pl
src/test/recovery/t/032_relfilenode_reuse.pl
src/test/recovery/t/037_invalid_database.pl
src/test/ssl/t/SSL/Server.pm
src/tools/ci/pg_ci_base.conf
I suspect in some of these cases using one of the new more granular
values would be appropriate? This could also serve as examples and
testing of the parameter itself.
On 20 May 2025, at 10:52, Peter Eisentraut <peter@eisentraut.org> wrote:
src/test/ssl/t/SSL/Server.pm
While I don't have an immediate usecase or test in mind, having the SSL tests
use log_conections=all could be handy when hacking on the TLS support.
--
Daniel Gustafsson
On Tue, May 20, 2025 at 4:52 AM Peter Eisentraut <peter@eisentraut.org>
wrote:
log_connections has been changed from a Boolean parameter to a string
one, but a number of places in the documentation and in various pieces
of test code still use the old values. I think it would be better if
they were adjusted to the new style.There are two places in doc/src/sgml/config.sgml where
log_connections=yes is used as an example. This is a relatively
prominent place, so it should not use deprecated values.
In earlier versions of my patch, I played around with replacing these
references in the docs. I ended up not doing it because I wasn't sure we
had consensus on deprecating the "on", "true", "yes" options and that we
would continue to support them indefinitely. Thinking about it now, by no
longer documenting "on" and "off", I was obviously deprecating them (not to
mention removing support for log_connections = "y", "ye", etc).
I'll write a patch to change these.
In src/backend/tcop/postgres.c, there is a call
SetConfigOption("log_connections", "true", context, source);
that could be adjusted.
Do you think the debug option should be 'all' or a list of the options
covered by "true" (which is a subset of 'all')?
Various uses in test code:
src/interfaces/libpq/t/005_negotiate_encryption.pl
src/test/authentication/t/001_password.pl
src/test/authentication/t/003_peer.pl
src/test/authentication/t/005_sspi.pl
src/test/authentication/t/007_pre_auth.pl
src/test/kerberos/t/001_auth.pl
src/test/ldap/t/001_auth.pl
src/test/ldap/t/002_bindpasswd.pl
src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
src/test/modules/oauth_validator/t/001_server.pl
src/test/modules/oauth_validator/t/002_client.pl
src/test/postmaster/t/002_connection_limits.pl
src/test/postmaster/t/003_start_stop.pl
src/test/recovery/t/013_crash_restart.pl
src/test/recovery/t/022_crash_temp_files.pl
src/test/recovery/t/032_relfilenode_reuse.pl
src/test/recovery/t/037_invalid_database.pl
src/test/ssl/t/SSL/Server.pm
src/tools/ci/pg_ci_base.confI suspect in some of these cases using one of the new more granular
values would be appropriate? This could also serve as examples and
testing of the parameter itself.
Yep, for these, some of them specifically parse and test output generated
by one or more of the log_connections options. In those cases, obviously
those options should be provided at a minimum. However, I assume, based on
how I use the logs from TAP tests, that when a test fails, folks use the
logging to understand more about what went wrong even when the log output
is not specifically parsed and tested against.
The reason I didn't change these was that I was unsure which of these tests
would want which options.
And I didn't want to risk increasing the volume of logging output by
replacing any of them with "all".
I can, of course, go through and make my best guess. Or I could just
replace them with the equivalent subset of options.
In the absence of clarity on this, I did add tests to 001_password.pl
specifically exercising and testing combinations of log_connections options.
- Melanie
On Tue, May 20, 2025 at 11:16 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:
In earlier versions of my patch, I played around with replacing these references in the docs. I ended up not doing it because I wasn't sure we had consensus on deprecating the "on", "true", "yes" options and that we would continue to support them indefinitely. Thinking about it now, by no longer documenting "on" and "off", I was obviously deprecating them (not to mention removing support for log_connections = "y", "ye", etc).
I'll write a patch to change these.
Attached is a patch that updates these as well as changes all usages
of log_connections in the tests. I made some judgment calls about
where we might want expanded or reduced log_connections aspects. As
such, the patch could use a once-over from someone else.
- Melanie
Attachments:
v1-0001-Replace-deprecated-log_connections-values-in-docs.patchtext/x-patch; charset=US-ASCII; name=v1-0001-Replace-deprecated-log_connections-values-in-docs.patchDownload
From e81b5ed2503c7a973397eea46ba16df2eb03f8f0 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Wed, 21 May 2025 15:26:42 -0400
Subject: [PATCH v1] Replace deprecated log_connections values in docs and
tests
9219093cab2607f modularized log_connections output to allow more
granular control over which aspects of connection establishment are
logged. It turned the boolean log_connections GUC into a list of strings
and deprecated previously supported boolean-like values on, off, true,
false, 1, 0, yes, and no. Those values still work, but they are
supported mainly for backwards compatability. As such, documented
examples of log_connections should not use the deprecated values.
Update references in the docs to deprecated log_connections values. Many
of the tests use log_connections. This commit also updates the tests to
use the new values of log_connections. In some of the tests, the updated
log_connections value covers a narrower set of aspects (e.g. the
'authentication' aspect in the tests in src/test/authentication and the
'receipt' aspect in src/test/postmaster). In other cases, the new value
for log_connections is a superset of the previous included aspects (e.g.
'all' in src/test/kerberos/t/001_auth.pl).
Author: Melanie Plageman <melanieplageman@gmail.com>
Reported-by: Peter Eisentraut <peter@eisentraut.org>
Discussion: https://postgr.es/m/e1586594-3b69-4aea-87ce-73a7488cdc97%40eisentraut.org
---
doc/src/sgml/config.sgml | 4 ++--
src/backend/tcop/postgres.c | 2 +-
src/interfaces/libpq/t/005_negotiate_encryption.pl | 2 +-
src/test/authentication/t/003_peer.pl | 2 +-
src/test/authentication/t/005_sspi.pl | 2 +-
src/test/authentication/t/007_pre_auth.pl | 2 +-
src/test/kerberos/t/001_auth.pl | 2 +-
src/test/ldap/t/001_auth.pl | 2 +-
src/test/ldap/t/002_bindpasswd.pl | 2 +-
.../modules/ldap_password_func/t/001_mutated_bindpasswd.pl | 2 +-
src/test/modules/oauth_validator/t/001_server.pl | 2 +-
src/test/modules/oauth_validator/t/002_client.pl | 2 +-
src/test/postmaster/t/002_connection_limits.pl | 2 +-
src/test/postmaster/t/003_start_stop.pl | 2 +-
src/test/recovery/t/013_crash_restart.pl | 2 +-
src/test/recovery/t/022_crash_temp_files.pl | 2 +-
src/test/recovery/t/032_relfilenode_reuse.pl | 2 +-
src/test/recovery/t/037_invalid_database.pl | 2 +-
src/test/ssl/t/SSL/Server.pm | 2 +-
src/tools/ci/pg_ci_base.conf | 2 +-
20 files changed, 21 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index aa8f47a1591..ca2a567b2b1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -140,7 +140,7 @@
An example of what this file might look like is:
<programlisting>
# This is a comment
-log_connections = yes
+log_connections = all
log_destination = 'syslog'
search_path = '"$user", public'
shared_buffers = 128MB
@@ -337,7 +337,7 @@ UPDATE pg_settings SET setting = reset_val WHERE name = 'configuration_parameter
<option>-c name=value</option> command-line parameter, or its equivalent
<option>--name=value</option> variation. For example,
<programlisting>
-postgres -c log_connections=yes --log-destination='syslog'
+postgres -c log_connections=all --log-destination='syslog'
</programlisting>
Settings provided in this way override those set via
<filename>postgresql.conf</filename> or <command>ALTER SYSTEM</command>,
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 1ae51b1b391..6fd47cefdb2 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3695,7 +3695,7 @@ set_debug_options(int debug_flag, GucContext context, GucSource source)
if (debug_flag >= 1 && context == PGC_POSTMASTER)
{
- SetConfigOption("log_connections", "true", context, source);
+ SetConfigOption("log_connections", "all", context, source);
SetConfigOption("log_disconnections", "true", context, source);
}
if (debug_flag >= 2)
diff --git a/src/interfaces/libpq/t/005_negotiate_encryption.pl b/src/interfaces/libpq/t/005_negotiate_encryption.pl
index f6a453c1b41..ac6d8bcb4a6 100644
--- a/src/interfaces/libpq/t/005_negotiate_encryption.pl
+++ b/src/interfaces/libpq/t/005_negotiate_encryption.pl
@@ -107,7 +107,7 @@ $node->append_conf(
listen_addresses = '$hostaddr'
# Capturing the EVENTS that occur during tests requires these settings
-log_connections = on
+log_connections = 'receipt,authentication,authorization'
log_disconnections = on
trace_connection_negotiation = on
lc_messages = 'C'
diff --git a/src/test/authentication/t/003_peer.pl b/src/test/authentication/t/003_peer.pl
index 2879800eacf..f2320b62c87 100644
--- a/src/test/authentication/t/003_peer.pl
+++ b/src/test/authentication/t/003_peer.pl
@@ -71,7 +71,7 @@ sub test_role
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = authentication\n");
# Needed to allow connect_fails to inspect postmaster log:
$node->append_conf('postgresql.conf', "log_min_messages = debug2");
$node->start;
diff --git a/src/test/authentication/t/005_sspi.pl b/src/test/authentication/t/005_sspi.pl
index b480b702590..cb3e169002f 100644
--- a/src/test/authentication/t/005_sspi.pl
+++ b/src/test/authentication/t/005_sspi.pl
@@ -18,7 +18,7 @@ if (!$windows_os || $use_unix_sockets)
# Initialize primary node
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = authentication\n");
$node->start;
my $huge_pages_status =
diff --git a/src/test/authentication/t/007_pre_auth.pl b/src/test/authentication/t/007_pre_auth.pl
index 12e40dc722c..4583ce387b0 100644
--- a/src/test/authentication/t/007_pre_auth.pl
+++ b/src/test/authentication/t/007_pre_auth.pl
@@ -20,7 +20,7 @@ my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
$node->append_conf(
'postgresql.conf', q[
-log_connections = on
+log_connections = authentication
]);
$node->start;
diff --git a/src/test/kerberos/t/001_auth.pl b/src/test/kerberos/t/001_auth.pl
index 2dc6bec9b89..b0be96f2beb 100644
--- a/src/test/kerberos/t/001_auth.pl
+++ b/src/test/kerberos/t/001_auth.pl
@@ -65,7 +65,7 @@ $node->append_conf(
'postgresql.conf', qq{
listen_addresses = '$hostaddr'
krb_server_keyfile = '$krb->{keytab}'
-log_connections = on
+log_connections = all
log_min_messages = debug2
lc_messages = 'C'
});
diff --git a/src/test/ldap/t/001_auth.pl b/src/test/ldap/t/001_auth.pl
index d1315ed5351..440c30b7ddd 100644
--- a/src/test/ldap/t/001_auth.pl
+++ b/src/test/ldap/t/001_auth.pl
@@ -47,7 +47,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = all\n");
# Needed to allow connect_fails to inspect postmaster log:
$node->append_conf('postgresql.conf', "log_min_messages = debug2");
$node->start;
diff --git a/src/test/ldap/t/002_bindpasswd.pl b/src/test/ldap/t/002_bindpasswd.pl
index f8beba2b279..642bb2d9a77 100644
--- a/src/test/ldap/t/002_bindpasswd.pl
+++ b/src/test/ldap/t/002_bindpasswd.pl
@@ -43,7 +43,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = all\n");
$node->start;
$node->safe_psql('postgres', 'CREATE USER test0;');
diff --git a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
index 9b062e1c800..6a0a15c242f 100644
--- a/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
+++ b/src/test/modules/ldap_password_func/t/001_mutated_bindpasswd.pl
@@ -42,7 +42,7 @@ note "setting up PostgreSQL instance";
my $node = PostgreSQL::Test::Cluster->new('node');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = 'receipt,authentication,authorization'\n");
$node->append_conf('postgresql.conf',
"shared_preload_libraries = 'ldap_password_func'");
$node->start;
diff --git a/src/test/modules/oauth_validator/t/001_server.pl b/src/test/modules/oauth_validator/t/001_server.pl
index 4f035417a40..bfc9dc3b542 100644
--- a/src/test/modules/oauth_validator/t/001_server.pl
+++ b/src/test/modules/oauth_validator/t/001_server.pl
@@ -45,7 +45,7 @@ if ($ENV{with_python} ne 'yes')
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = all\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
# Needed to allow connect_fails to inspect postmaster log:
diff --git a/src/test/modules/oauth_validator/t/002_client.pl b/src/test/modules/oauth_validator/t/002_client.pl
index 21d4acc1926..aac0220d215 100644
--- a/src/test/modules/oauth_validator/t/002_client.pl
+++ b/src/test/modules/oauth_validator/t/002_client.pl
@@ -26,7 +26,7 @@ if (!$ENV{PG_TEST_EXTRA} || $ENV{PG_TEST_EXTRA} !~ /\boauth\b/)
my $node = PostgreSQL::Test::Cluster->new('primary');
$node->init;
-$node->append_conf('postgresql.conf', "log_connections = on\n");
+$node->append_conf('postgresql.conf', "log_connections = all\n");
$node->append_conf('postgresql.conf',
"oauth_validator_libraries = 'validator'\n");
$node->start;
diff --git a/src/test/postmaster/t/002_connection_limits.pl b/src/test/postmaster/t/002_connection_limits.pl
index 325a00efd47..274aec2d43d 100644
--- a/src/test/postmaster/t/002_connection_limits.pl
+++ b/src/test/postmaster/t/002_connection_limits.pl
@@ -20,7 +20,7 @@ $node->init(
$node->append_conf('postgresql.conf', "max_connections = 6");
$node->append_conf('postgresql.conf', "reserved_connections = 2");
$node->append_conf('postgresql.conf', "superuser_reserved_connections = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'receipt,authentication,authorization'");
$node->append_conf('postgresql.conf', "log_min_messages=debug2");
$node->start;
diff --git a/src/test/postmaster/t/003_start_stop.pl b/src/test/postmaster/t/003_start_stop.pl
index 4dc394139d9..7af18e59608 100644
--- a/src/test/postmaster/t/003_start_stop.pl
+++ b/src/test/postmaster/t/003_start_stop.pl
@@ -33,7 +33,7 @@ $node->append_conf('postgresql.conf', "max_connections = 5");
$node->append_conf('postgresql.conf', "max_wal_senders = 0");
$node->append_conf('postgresql.conf', "autovacuum_max_workers = 1");
$node->append_conf('postgresql.conf', "max_worker_processes = 1");
-$node->append_conf('postgresql.conf', "log_connections = on");
+$node->append_conf('postgresql.conf', "log_connections = 'receipt,authentication,authorization'");
$node->append_conf('postgresql.conf', "log_min_messages = debug2");
$node->append_conf('postgresql.conf',
"authentication_timeout = '$authentication_timeout s'");
diff --git a/src/test/recovery/t/013_crash_restart.pl b/src/test/recovery/t/013_crash_restart.pl
index 4e60806563f..debfa635c36 100644
--- a/src/test/recovery/t/013_crash_restart.pl
+++ b/src/test/recovery/t/013_crash_restart.pl
@@ -27,7 +27,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET restart_after_crash = 1;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = receipt;
SELECT pg_reload_conf();]);
# Run psql, keeping session alive, so we have an alive backend to kill.
diff --git a/src/test/recovery/t/022_crash_temp_files.pl b/src/test/recovery/t/022_crash_temp_files.pl
index 50def031c96..0b68860bd3e 100644
--- a/src/test/recovery/t/022_crash_temp_files.pl
+++ b/src/test/recovery/t/022_crash_temp_files.pl
@@ -26,7 +26,7 @@ $node->start();
$node->safe_psql(
'postgres',
q[ALTER SYSTEM SET remove_temp_files_after_crash = on;
- ALTER SYSTEM SET log_connections = 1;
+ ALTER SYSTEM SET log_connections = receipt;
ALTER SYSTEM SET work_mem = '64kB';
ALTER SYSTEM SET restart_after_crash = on;
SELECT pg_reload_conf();]);
diff --git a/src/test/recovery/t/032_relfilenode_reuse.pl b/src/test/recovery/t/032_relfilenode_reuse.pl
index 492ef115ba4..0c44883cc34 100644
--- a/src/test/recovery/t/032_relfilenode_reuse.pl
+++ b/src/test/recovery/t/032_relfilenode_reuse.pl
@@ -14,7 +14,7 @@ $node_primary->init(allows_streaming => 1);
$node_primary->append_conf(
'postgresql.conf', q[
allow_in_place_tablespaces = true
-log_connections=on
+log_connections=receipt
# to avoid "repairing" corruption
full_page_writes=off
log_min_messages=debug2
diff --git a/src/test/recovery/t/037_invalid_database.pl b/src/test/recovery/t/037_invalid_database.pl
index bdf39397397..dc52c55c7af 100644
--- a/src/test/recovery/t/037_invalid_database.pl
+++ b/src/test/recovery/t/037_invalid_database.pl
@@ -15,7 +15,7 @@ $node->append_conf(
autovacuum = off
max_prepared_transactions=5
log_min_duration_statement=0
-log_connections=on
+log_connections=receipt
log_disconnections=on
));
diff --git a/src/test/ssl/t/SSL/Server.pm b/src/test/ssl/t/SSL/Server.pm
index 33975b28e8c..96f0f201e9c 100644
--- a/src/test/ssl/t/SSL/Server.pm
+++ b/src/test/ssl/t/SSL/Server.pm
@@ -200,7 +200,7 @@ sub configure_test_server_for_ssl
$node->append_conf(
'postgresql.conf', <<EOF
fsync=off
-log_connections=on
+log_connections=all
log_hostname=on
listen_addresses='$serverhost'
log_statement=all
diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf
index d8faa9c26c1..9cec5c2910d 100644
--- a/src/tools/ci/pg_ci_base.conf
+++ b/src/tools/ci/pg_ci_base.conf
@@ -8,7 +8,7 @@ max_prepared_transactions = 10
# Settings that make logs more useful
log_autovacuum_min_duration = 0
log_checkpoints = true
-log_connections = true
+log_connections = all
log_disconnections = true
log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] '
log_lock_waits = true
--
2.34.1
On Wed, May 21, 2025 at 12:54 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:
Attached is a patch that updates these as well as changes all usages
of log_connections in the tests. I made some judgment calls about
where we might want expanded or reduced log_connections aspects. As
such, the patch could use a once-over from someone else.
I took a quick look at the authentication and oauth_validator changes.
Maybe 007_pre_auth.pl should include `receipt`, to make it easier to
debug pre-auth hangs using the logs? Other than that, the changes
looked reasonable to me.
--Jacob
On 21.05.25 21:53, Melanie Plageman wrote:
On Tue, May 20, 2025 at 11:16 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:In earlier versions of my patch, I played around with replacing these references in the docs. I ended up not doing it because I wasn't sure we had consensus on deprecating the "on", "true", "yes" options and that we would continue to support them indefinitely. Thinking about it now, by no longer documenting "on" and "off", I was obviously deprecating them (not to mention removing support for log_connections = "y", "ye", etc).
I'll write a patch to change these.
Attached is a patch that updates these as well as changes all usages
of log_connections in the tests. I made some judgment calls about
where we might want expanded or reduced log_connections aspects. As
such, the patch could use a once-over from someone else.
This looks good to me.
On 20.05.25 17:16, Melanie Plageman wrote:
In src/backend/tcop/postgres.c, there is a call
SetConfigOption("log_connections", "true", context, source);
that could be adjusted.
Do you think the debug option should be 'all' or a list of the options
covered by "true" (which is a subset of 'all')?
I think the spirit of this debug option is to show lots of things, so
mapping it to 'all' makes sense to me.
On Wed, May 21, 2025 at 5:20 PM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
I took a quick look at the authentication and oauth_validator changes.
Maybe 007_pre_auth.pl should include `receipt`, to make it easier to
debug pre-auth hangs using the logs? Other than that, the changes
looked reasonable to me.
Cool, I updated that one (had to review my good ol' perl single
quoting rules to see if q[] still worked there but I think we're safe
:)). Thanks for taking a look.
Thanks also to Peter E for the report and review. I've pushed this.
- Melanie