Improve handling of parameter differences in physical replication

From 3fb5c70b4b7e132cb75271bbd70c4a1523a72e86 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 11 Mar 2020 19:25:44 +0100
Subject: [PATCH v2] Improve handling of parameter differences in physical
 replication

When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record.  The standby then checks whether its own settings are at least
as big as the ones on the primary.  If not, the standby shuts down
with a fatal error.

The correspondence of settings between primary and standby is required
because those settings influence certain shared memory sizings that
are required for processing WAL records that the primary might send.
For example, if the primary sends a prepared transaction, the standby
must have had max_prepared_transaction set appropriately or it won't
be able to process those WAL records.

However, fatally shutting down the standby immediately upon receipt of
the parameter change record might be a bit of an overreaction.  The
resources related to those settings are not required immediately at
that point, and might never be required if the activity on the primary
does not exhaust all those resources.  If we just let the standby roll
on with recovery, it will eventually produce an appropriate error when
those resources are used.

So this patch relaxes this a bit.  Upon receipt of
XLOG_PARAMETER_CHANGE, we still check the settings but only issue a
warning and set a global flag if there is a problem.  Then when we
actually hit the resource issue and the flag was set, we issue another
warning message with relevant information.  Additionally, at that
point we pause recovery, so a hot standby remains usable.
---
 doc/src/sgml/high-availability.sgml   | 48 +++++++++++++-----
 src/backend/access/transam/twophase.c |  3 ++
 src/backend/access/transam/xlog.c     | 72 +++++++++++++++++++++++----
 src/backend/storage/ipc/procarray.c   |  9 +++-
 src/backend/storage/lmgr/lock.c       | 10 ++++
 src/include/access/xlog.h             |  1 +
 6 files changed, 120 insertions(+), 23 deletions(-)

diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index bc4d98fe03..c5a9f9d5df 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -2148,18 +2148,14 @@ <title>Administrator's Overview</title>
    </para>
 
    <para>
-    The setting of some parameters on the standby will need reconfiguration
-    if they have been changed on the primary. For these parameters,
-    the value on the standby must
-    be equal to or greater than the value on the primary.
-    Therefore, if you want to increase these values, you should do so on all
-    standby servers first, before applying the changes to the primary server.
-    Conversely, if you want to decrease these values, you should do so on the
-    primary server first, before applying the changes to all standby servers.
-    If these parameters
-    are not set high enough then the standby will refuse to start.
-    Higher values can then be supplied and the server
-    restarted to begin recovery again.  These parameters are:
+    The settings of some parameters determine the size of shared memory for
+    tracking transaction IDs, locks, and prepared transactions.  These shared
+    memory structures should be no smaller on a standby than on the primary.
+    Otherwise, it could happen that the standby runs out of shared memory
+    during recovery.  For example, if the primary uses a prepared transaction
+    but the standby did not allocate any shared memory for tracking prepared
+    transactions, then recovery will abort and cannot continue until the
+    standby's configuration is changed.  The parameters affected are:
 
       <itemizedlist>
        <listitem>
@@ -2188,6 +2184,34 @@ <title>Administrator's Overview</title>
         </para>
        </listitem>
       </itemizedlist>
+
+    The easiest way to ensure this does not become a problem is to have these
+    parameters set on the standbys to values equal to or greater than on the
+    primary.  Therefore, if you want to increase these values, you should do
+    so on all standby servers first, before applying the changes to the
+    primary server.  Conversely, if you want to decrease these values, you
+    should do so on the primary server first, before applying the changes to
+    all standby servers.  The WAL tracks changes to these parameters on the
+    primary, and if a standby processes WAL that indicates that the current
+    value on the primary is higher than its own value, it will log a warning, for example:
+<screen>
+WARNING:  insufficient setting for parameter max_connections
+DETAIL:  max_connections = 80 is a lower setting than on the master server (where its value was 100).
+HINT:  Change parameters and restart the server, or there may be resource exhaustion errors sooner or later.
+</screen>
+    Recovery will continue but could abort at any time thereafter.  (It could
+    also never end up failing if the activity on the primary does not actually
+    require the full extent of the allocated shared memory resources.)  If
+    recovery reaches a point where it cannot continue due to lack of shared
+    memory, recovery will pause and another warning will be logged, for example:
+<screen>
+WARNING:  recovery paused because of insufficient parameter settings
+DETAIL:  See earlier in the log about which settings are insufficient.
+HINT:  Recovery cannot continue unless the configuration is changed and the server restarted.
+</screen>
+    This warning will repeated once a minute.  At that point, the settings on
+    the standby need to be updated and the instance restarted before recovery
+    can continue.
    </para>
 
    <para>
diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c
index 5adf956f41..ce35f15f34 100644
--- a/src/backend/access/transam/twophase.c
+++ b/src/backend/access/transam/twophase.c
@@ -2360,11 +2360,14 @@ PrepareRedoAdd(char *buf, XLogRecPtr start_lsn,
 
 	/* Get a free gxact from the freelist */
 	if (TwoPhaseState->freeGXacts == NULL)
+	{
+		StandbyParamErrorPauseRecovery();
 		ereport(ERROR,
 				(errcode(ERRCODE_OUT_OF_MEMORY),
 				 errmsg("maximum number of prepared transactions reached"),
 				 errhint("Increase max_prepared_transactions (currently %d).",
 						 max_prepared_xacts)));
+	}
 	gxact = TwoPhaseState->freeGXacts;
 	TwoPhaseState->freeGXacts = gxact->next;
 
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 614a25242b..3e825a2414 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -258,6 +258,8 @@ bool		InArchiveRecovery = false;
 static bool standby_signal_file_found = false;
 static bool recovery_signal_file_found = false;
 
+static bool need_restart_for_parameter_values = false;
+
 /* Was the last xlog file restored from archive, or local? */
 static bool restoredFromArchive = false;
 
@@ -5971,6 +5973,52 @@ SetRecoveryPause(bool recoveryPause)
 	SpinLockRelease(&XLogCtl->info_lck);
 }
 
+/*
+ * If in hot standby, pause recovery because of a parameter conflict.
+ *
+ * Similar to recoveryPausesHere() but with a different messaging.  The user
+ * is expected to make the parameter change and restart the server.  If they
+ * just unpause recovery, they will then run into whatever error change is
+ * after this function call for the non-hot-standby case.
+ *
+ * We intentionally do not give advice about specific parameters or values
+ * here because it might be misleading.  For example, if we run out of lock
+ * space, then in the single-server case we would recommend raising
+ * max_locks_per_transaction, but in recovery it could equally be the case
+ * that max_connections is out of sync with the primary.  If we get here, we
+ * have already logged any parameter discrepancy in
+ * RecoveryRequiresIntParameter(), so users can go back to that and get
+ * concrete and accurate information.
+ */
+void
+StandbyParamErrorPauseRecovery(void)
+{
+	TimestampTz last_warning = 0;
+
+	if (!AmStartupProcess() || !need_restart_for_parameter_values)
+		return;
+
+	SetRecoveryPause(true);
+
+	do
+	{
+		TimestampTz now = GetCurrentTimestamp();
+
+		if (TimestampDifferenceExceeds(last_warning, now, 60000))
+		{
+			ereport(WARNING,
+					(errmsg("recovery paused because of insufficient parameter settings"),
+					 errdetail("See earlier in the log about which settings are insufficient."),
+					 errhint("Recovery cannot continue unless the configuration is changed and the server restarted.")));
+			last_warning = now;
+		}
+
+		pg_usleep(1000000L);    /* 1000 ms */
+		HandleStartupProcInterrupts();
+	}
+	while (RecoveryIsPaused());
+}
+
 /*
  * When recovery_min_apply_delay is set, we wait long enough to make sure
  * certain record types are applied at least that interval behind the master.
@@ -6150,16 +6198,20 @@ GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream)
  * Note that text field supplied is a parameter name and does not require
  * translation
  */
-#define RecoveryRequiresIntParameter(param_name, currValue, minValue) \
-do { \
-	if ((currValue) < (minValue)) \
-		ereport(ERROR, \
-				(errcode(ERRCODE_INVALID_PARAMETER_VALUE), \
-				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)", \
-						param_name, \
-						currValue, \
-						minValue))); \
-} while(0)
+static void
+RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
+{
+	if (currValue < minValue)
+	{
+		ereport(WARNING,
+				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+				 errmsg("insufficient setting for parameter %s", param_name),
+				 errdetail("%s = %d is a lower setting than on the master server (where its value was %d).",
+						   param_name, currValue, minValue),
+				 errhint("Change parameters and restart the server, or there may be resource exhaustion errors sooner or later.")));
+		need_restart_for_parameter_values = true;
+	}
+}
 
 /*
  * Check to see if required parameters are set high enough on this server
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index f45a619deb..cfb88db4a4 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -3654,7 +3654,14 @@ KnownAssignedXidsAdd(TransactionId from_xid, TransactionId to_xid,
 		 * If it still won't fit then we're out of memory
 		 */
 		if (head + nxids > pArray->maxKnownAssignedXids)
-			elog(ERROR, "too many KnownAssignedXids");
+		{
+			StandbyParamErrorPauseRecovery();
+			ereport(ERROR,
+					(errcode(ERRCODE_OUT_OF_MEMORY),
+					 errmsg("out of shared memory"),
+					 errdetail("There are no more KnownAssignedXids slots."),
+					 errhint("You might need to increase max_connections.")));
+		}
 	}
 
 	/* Now we can insert the xids into the space starting at head */
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 56dba09299..cbcc7611f4 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -924,10 +924,13 @@ LockAcquireExtended(const LOCKTAG *locktag,
 			if (locallockp)
 				*locallockp = NULL;
 			if (reportMemoryError)
+			{
+				StandbyParamErrorPauseRecovery();
 				ereport(ERROR,
 						(errcode(ERRCODE_OUT_OF_MEMORY),
 						 errmsg("out of shared memory"),
 						 errhint("You might need to increase max_locks_per_transaction.")));
+			}
 			else
 				return LOCKACQUIRE_NOT_AVAIL;
 		}
@@ -962,10 +965,13 @@ LockAcquireExtended(const LOCKTAG *locktag,
 		if (locallockp)
 			*locallockp = NULL;
 		if (reportMemoryError)
+		{
+			StandbyParamErrorPauseRecovery();
 			ereport(ERROR,
 					(errcode(ERRCODE_OUT_OF_MEMORY),
 					 errmsg("out of shared memory"),
 					 errhint("You might need to increase max_locks_per_transaction.")));
+		}
 		else
 			return LOCKACQUIRE_NOT_AVAIL;
 	}
@@ -2747,6 +2753,7 @@ FastPathGetRelationLockEntry(LOCALLOCK *locallock)
 		{
 			LWLockRelease(partitionLock);
 			LWLockRelease(&MyProc->backendLock);
+			StandbyParamErrorPauseRecovery();
 			ereport(ERROR,
 					(errcode(ERRCODE_OUT_OF_MEMORY),
 					 errmsg("out of shared memory"),
@@ -4077,6 +4084,7 @@ lock_twophase_recover(TransactionId xid, uint16 info,
 	if (!lock)
 	{
 		LWLockRelease(partitionLock);
+		StandbyParamErrorPauseRecovery();
 		ereport(ERROR,
 				(errcode(ERRCODE_OUT_OF_MEMORY),
 				 errmsg("out of shared memory"),
@@ -4142,6 +4150,7 @@ lock_twophase_recover(TransactionId xid, uint16 info,
 				elog(PANIC, "lock table corrupted");
 		}
 		LWLockRelease(partitionLock);
+		StandbyParamErrorPauseRecovery();
 		ereport(ERROR,
 				(errcode(ERRCODE_OUT_OF_MEMORY),
 				 errmsg("out of shared memory"),
@@ -4434,6 +4443,7 @@ VirtualXactLock(VirtualTransactionId vxid, bool wait)
 		{
 			LWLockRelease(partitionLock);
 			LWLockRelease(&proc->backendLock);
+			StandbyParamErrorPauseRecovery();
 			ereport(ERROR,
 					(errcode(ERRCODE_OUT_OF_MEMORY),
 					 errmsg("out of shared memory"),
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033fc20..fae19a9d26 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -286,6 +286,7 @@ extern XLogRecPtr GetXLogInsertRecPtr(void);
 extern XLogRecPtr GetXLogWriteRecPtr(void);
 extern bool RecoveryIsPaused(void);
 extern void SetRecoveryPause(bool recoveryPause);
+extern void StandbyParamErrorPauseRecovery(void);
 extern TimestampTz GetLatestXTime(void);
 extern TimestampTz GetCurrentChunkReplayStartTime(void);
 
-- 
2.25.0

From 2d3ffb221ae9418ebd7e1fc7fb771213014ca216 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 08:49:42 +0200
Subject: [PATCH v3 1/2] Replace a macro by a function

Using a macro is ugly and not justified here.
---
 src/backend/access/transam/xlog.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index a1256a103b..dcec2111b3 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6232,16 +6232,17 @@ GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream)
  * Note that text field supplied is a parameter name and does not require
  * translation
  */
-#define RecoveryRequiresIntParameter(param_name, currValue, minValue) \
-do { \
-	if ((currValue) < (minValue)) \
-		ereport(ERROR, \
-				(errcode(ERRCODE_INVALID_PARAMETER_VALUE), \
-				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)", \
-						param_name, \
-						currValue, \
-						minValue))); \
-} while(0)
+static void
+RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
+{
+	if (currValue < minValue)
+		ereport(ERROR,
+				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",
+						param_name,
+						currValue,
+						minValue)));
+}
 
 /*
  * Check to see if required parameters are set high enough on this server
-- 
2.27.0

From 18740a021a90207423032cb7201f54650b81c15a Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 09:18:21 +0200
Subject: [PATCH v3 2/2] Pause recovery for insufficient parameter settings

When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record.  The standby then checks whether its own settings are at least
as big as the ones on the primary.  If not, the standby shuts down
with a fatal error.

This patch changes this behavior to pause recovery at that point
instead.  That allows read traffic on the standby to continue while
database administrators figure out next steps.  When recovery is
unpaused, the server shuts down (as before).  The idea is to fix the
parameters while recovery is paused and then restart when there is a
maintenance window.
---
 doc/src/sgml/high-availability.sgml | 47 +++++++++++++++++++++--------
 src/backend/access/transam/xlog.c   | 17 ++++++++++-
 2 files changed, 51 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 65c3fc62a9..d32ec6d9b3 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -2159,18 +2159,14 @@ <title>Administrator's Overview</title>
    </para>
 
    <para>
-    The setting of some parameters on the standby will need reconfiguration
-    if they have been changed on the primary. For these parameters,
-    the value on the standby must
-    be equal to or greater than the value on the primary.
-    Therefore, if you want to increase these values, you should do so on all
-    standby servers first, before applying the changes to the primary server.
-    Conversely, if you want to decrease these values, you should do so on the
-    primary server first, before applying the changes to all standby servers.
-    If these parameters
-    are not set high enough then the standby will refuse to start.
-    Higher values can then be supplied and the server
-    restarted to begin recovery again.  These parameters are:
+    The settings of some parameters determine the size of shared memory for
+    tracking transaction IDs, locks, and prepared transactions.  These shared
+    memory structures must be no smaller on a standby than on the primary in
+    order to ensure that the standby does not run out of shared memory during
+    recovery.  For example, if the primary had used a prepared transaction but
+    the standby had not allocated any shared memory for tracking prepared
+    transactions, then recovery could not continue until the standby's
+    configuration is changed.  The parameters affected are:
 
       <itemizedlist>
        <listitem>
@@ -2199,6 +2195,33 @@ <title>Administrator's Overview</title>
         </para>
        </listitem>
       </itemizedlist>
+
+    The easiest way to ensure this does not become a problem is to have these
+    parameters set on the standbys to values equal to or greater than on the
+    primary.  Therefore, if you want to increase these values, you should do
+    so on all standby servers first, before applying the changes to the
+    primary server.  Conversely, if you want to decrease these values, you
+    should do so on the primary server first, before applying the changes to
+    all standby servers.  Keep in mind that when a standby is promoted, it
+    becomes the new reference for the required parameter settings for the
+    standbys that follow it.  Therefore, to avoid this becoming a problem
+    during a switchover or failover, it is recommended to keep these settings
+    the same on all standby servers.
+   </para>
+
+   <para>
+    The WAL tracks changes to these parameters on the
+    primary, and if a standby processes WAL that indicates that the current
+    value on the primary is higher than its own value, it will log a warning
+    and pause recovery, for example:
+<screen>
+WARNING:  hot standby is not possible because max_connections = 80 is a lower setting than on the master server (its value was 100)
+LOG:  recovery will be paused
+DETAIL:  If recovery is unpaused, the server will shut down.
+HINT:  You can then restart the server after making the necessary configuration changes.
+</screen>
+    At that point, the settings on the standby need to be updated and the
+    instance restarted before recovery can continue.
    </para>
 
    <para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index dcec2111b3..e0e22e26f4 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6236,12 +6236,27 @@ static void
 RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
 {
 	if (currValue < minValue)
-		ereport(ERROR,
+	{
+		ereport(WARNING,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
 				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the master server (its value was %d)",
 						param_name,
 						currValue,
 						minValue)));
+		ereport(LOG,
+				(errmsg("recovery will be paused"),
+				 errdetail("If recovery is unpaused, the server will shut down."),
+				 errhint("You can then restart the server after making the necessary configuration changes.")));
+		SetRecoveryPause(true);
+		while (RecoveryIsPaused())
+		{
+			pg_usleep(1000000L);	/* 1000 ms */
+			HandleStartupProcInterrupts();
+		}
+		ereport(FATAL,
+				(errmsg("recovery aborted because of insufficient parameter settings"),
+				 errhint("You can restart the server after making the necessary configuration changes.")));
+	}
 }
 
 /*
-- 
2.27.0

From 6dfa5445eed5c4bf601577df1a874bf10b0e562a Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 08:49:42 +0200
Subject: [PATCH v4 1/2] Replace a macro by a function

Using a macro is ugly and not justified here.
---
 src/backend/access/transam/xlog.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 0a97b1d37f..b9d011fa8d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6235,16 +6235,17 @@ GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream)
  * Note that text field supplied is a parameter name and does not require
  * translation
  */
-#define RecoveryRequiresIntParameter(param_name, currValue, minValue) \
-do { \
-	if ((currValue) < (minValue)) \
-		ereport(ERROR, \
-				(errcode(ERRCODE_INVALID_PARAMETER_VALUE), \
-				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the primary server (its value was %d)", \
-						param_name, \
-						currValue, \
-						minValue))); \
-} while(0)
+static void
+RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
+{
+	if (currValue < minValue)
+		ereport(ERROR,
+				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the primary server (its value was %d)",
+						param_name,
+						currValue,
+						minValue)));
+}
 
 /*
  * Check to see if required parameters are set high enough on this server
-- 
2.27.0

From 4b5812c41fe581042249a1af43ac96df75f14a23 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Wed, 24 Jun 2020 09:18:21 +0200
Subject: [PATCH v4 2/2] Pause recovery for insufficient parameter settings

When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record.  The standby then checks whether its own settings are at least
as big as the ones on the primary.  If not, the standby shuts down
with a fatal error.

This patch changes this behavior to pause recovery at that point
instead.  That allows read traffic on the standby to continue while
database administrators figure out next steps.  When recovery is
unpaused, the server shuts down (as before).  The idea is to fix the
parameters while recovery is paused and then restart when there is a
maintenance window.
---
 doc/src/sgml/high-availability.sgml | 47 +++++++++++++++++++++--------
 src/backend/access/transam/xlog.c   | 17 ++++++++++-
 2 files changed, 51 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 6a9184f314..5aa8eeced2 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -2158,18 +2158,14 @@ <title>Administrator's Overview</title>
    </para>
 
    <para>
-    The setting of some parameters on the standby will need reconfiguration
-    if they have been changed on the primary. For these parameters,
-    the value on the standby must
-    be equal to or greater than the value on the primary.
-    Therefore, if you want to increase these values, you should do so on all
-    standby servers first, before applying the changes to the primary server.
-    Conversely, if you want to decrease these values, you should do so on the
-    primary server first, before applying the changes to all standby servers.
-    If these parameters
-    are not set high enough then the standby will refuse to start.
-    Higher values can then be supplied and the server
-    restarted to begin recovery again.  These parameters are:
+    The settings of some parameters determine the size of shared memory for
+    tracking transaction IDs, locks, and prepared transactions.  These shared
+    memory structures must be no smaller on a standby than on the primary in
+    order to ensure that the standby does not run out of shared memory during
+    recovery.  For example, if the primary had used a prepared transaction but
+    the standby had not allocated any shared memory for tracking prepared
+    transactions, then recovery could not continue until the standby's
+    configuration is changed.  The parameters affected are:
 
       <itemizedlist>
        <listitem>
@@ -2198,6 +2194,33 @@ <title>Administrator's Overview</title>
         </para>
        </listitem>
       </itemizedlist>
+
+    The easiest way to ensure this does not become a problem is to have these
+    parameters set on the standbys to values equal to or greater than on the
+    primary.  Therefore, if you want to increase these values, you should do
+    so on all standby servers first, before applying the changes to the
+    primary server.  Conversely, if you want to decrease these values, you
+    should do so on the primary server first, before applying the changes to
+    all standby servers.  Keep in mind that when a standby is promoted, it
+    becomes the new reference for the required parameter settings for the
+    standbys that follow it.  Therefore, to avoid this becoming a problem
+    during a switchover or failover, it is recommended to keep these settings
+    the same on all standby servers.
+   </para>
+
+   <para>
+    The WAL tracks changes to these parameters on the
+    primary, and if a standby processes WAL that indicates that the current
+    value on the primary is higher than its own value, it will log a warning
+    and pause recovery, for example:
+<screen>
+WARNING:  hot standby is not possible because max_connections = 80 is a lower setting than on the master server (its value was 100)
+LOG:  recovery will be paused
+DETAIL:  If recovery is unpaused, the server will shut down.
+HINT:  You can then restart the server after making the necessary configuration changes.
+</screen>
+    At that point, the settings on the standby need to be updated and the
+    instance restarted before recovery can continue.
    </para>
 
    <para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index b9d011fa8d..d5ae40cdf7 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6239,12 +6239,27 @@ static void
 RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
 {
 	if (currValue < minValue)
-		ereport(ERROR,
+	{
+		ereport(WARNING,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
 				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the primary server (its value was %d)",
 						param_name,
 						currValue,
 						minValue)));
+		ereport(LOG,
+				(errmsg("recovery will be paused"),
+				 errdetail("If recovery is unpaused, the server will shut down."),
+				 errhint("You can then restart the server after making the necessary configuration changes.")));
+		SetRecoveryPause(true);
+		while (RecoveryIsPaused())
+		{
+			pg_usleep(1000000L);	/* 1000 ms */
+			HandleStartupProcInterrupts();
+		}
+		ereport(FATAL,
+				(errmsg("recovery aborted because of insufficient parameter settings"),
+				 errhint("You can restart the server after making the necessary configuration changes.")));
+	}
 }
 
 /*
-- 
2.27.0

From 99724e2ee14b5f3ec926c7afdc056863a7e2294f Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Fri, 20 Nov 2020 13:39:09 +0100
Subject: [PATCH v5] Pause recovery for insufficient parameter settings

When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record.  The standby then checks whether its own settings are at least
as big as the ones on the primary.  If not, the standby shuts down
with a fatal error.

This patch changes this behavior to pause recovery at that point
instead.  That allows read traffic on the standby to continue while
database administrators figure out next steps.  When recovery is
unpaused, the server shuts down (as before).  The idea is to fix the
parameters while recovery is paused and then restart when there is a
maintenance window.

Discussion: https://www.postgresql.org/message-id/flat/4ad69a4c-cc9b-0dfe-0352-8b1b0cd36c7b@2ndquadrant.com
---
 doc/src/sgml/high-availability.sgml | 48 +++++++++++++++++++++--------
 src/backend/access/transam/xlog.c   | 38 ++++++++++++++++++++---
 2 files changed, 69 insertions(+), 17 deletions(-)

diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 19d7bd2b28..e9a30dd88b 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -2120,18 +2120,14 @@ <title>Administrator's Overview</title>
    </para>
 
    <para>
-    The setting of some parameters on the standby will need reconfiguration
-    if they have been changed on the primary. For these parameters,
-    the value on the standby must
-    be equal to or greater than the value on the primary.
-    Therefore, if you want to increase these values, you should do so on all
-    standby servers first, before applying the changes to the primary server.
-    Conversely, if you want to decrease these values, you should do so on the
-    primary server first, before applying the changes to all standby servers.
-    If these parameters
-    are not set high enough then the standby will refuse to start.
-    Higher values can then be supplied and the server
-    restarted to begin recovery again.  These parameters are:
+    The settings of some parameters determine the size of shared memory for
+    tracking transaction IDs, locks, and prepared transactions.  These shared
+    memory structures must be no smaller on a standby than on the primary in
+    order to ensure that the standby does not run out of shared memory during
+    recovery.  For example, if the primary had used a prepared transaction but
+    the standby had not allocated any shared memory for tracking prepared
+    transactions, then recovery could not continue until the standby's
+    configuration is changed.  The parameters affected are:
 
       <itemizedlist>
        <listitem>
@@ -2160,6 +2156,34 @@ <title>Administrator's Overview</title>
         </para>
        </listitem>
       </itemizedlist>
+
+    The easiest way to ensure this does not become a problem is to have these
+    parameters set on the standbys to values equal to or greater than on the
+    primary.  Therefore, if you want to increase these values, you should do
+    so on all standby servers first, before applying the changes to the
+    primary server.  Conversely, if you want to decrease these values, you
+    should do so on the primary server first, before applying the changes to
+    all standby servers.  Keep in mind that when a standby is promoted, it
+    becomes the new reference for the required parameter settings for the
+    standbys that follow it.  Therefore, to avoid this becoming a problem
+    during a switchover or failover, it is recommended to keep these settings
+    the same on all standby servers.
+   </para>
+
+   <para>
+    The WAL tracks changes to these parameters on the
+    primary, and if a standby processes WAL that indicates that the current
+    value on the primary is higher than its own value, it will log a warning
+    and pause recovery, for example:
+<screen>
+WARNING:  hot standby is not possible because of insufficient parameter settings
+DETAIL:  max_connections = 80 is a lower setting than on the primary server, where its value was 100.
+LOG:  recovery has paused
+DETAIL:  If recovery is unpaused, the server will shut down.
+HINT:  You can then restart the server after making the necessary configuration changes.
+</screen>
+    At that point, the settings on the standby need to be updated and the
+    instance restarted before recovery can continue.
    </para>
 
    <para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13f1d8c3dc..a3519d50e4 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6232,12 +6232,40 @@ static void
 RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
 {
 	if (currValue < minValue)
-		ereport(ERROR,
+	{
+		ereport(WARNING,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
-				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the primary server (its value was %d)",
-						param_name,
-						currValue,
-						minValue)));
+				 errmsg("hot standby is not possible because of insufficient parameter settings"),
+				 errdetail("%s = %d is a lower setting than on the primary server, where its value was %d.",
+						   param_name,
+						   currValue,
+						   minValue)));
+
+		SetRecoveryPause(true);
+
+		ereport(LOG,
+				(errmsg("recovery has paused"),
+				 errdetail("If recovery is unpaused, the server will shut down."),
+				 errhint("You can then restart the server after making the necessary configuration changes.")));
+
+		while (RecoveryIsPaused())
+		{
+			HandleStartupProcInterrupts();
+			pgstat_report_wait_start(WAIT_EVENT_RECOVERY_PAUSE);
+			pg_usleep(1000000L);	/* 1000 ms */
+			pgstat_report_wait_end();
+		}
+
+		ereport(FATAL,
+				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+				 errmsg("recovery aborted because of insufficient parameter settings"),
+				 /* Repeat the detail from above so it's easy to find in the log. */
+				 errdetail("%s = %d is a lower setting than on the primary server, where its value was %d.",
+						   param_name,
+						   currValue,
+						   minValue),
+				 errhint("You can restart the server after making the necessary configuration changes.")));
+	}
 }
 
 /*
-- 
2.29.2

From 2d93c2918af7ff186aa4a6a2327286b6eb1a2859 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Mon, 30 Nov 2020 19:21:40 +0100
Subject: [PATCH v6] Pause recovery for insufficient parameter settings

When certain parameters are changed on a physical replication primary,
this is communicated to standbys using the XLOG_PARAMETER_CHANGE WAL
record.  The standby then checks whether its own settings are at least
as big as the ones on the primary.  If not, the standby shuts down
with a fatal error.

This patch changes this behavior for hot standbys to pause recovery at
that point instead.  That allows read traffic on the standby to
continue while database administrators figure out next steps.  When
recovery is unpaused, the server shuts down (as before).  The idea is
to fix the parameters while recovery is paused and then restart when
there is a maintenance window.

Discussion: https://www.postgresql.org/message-id/flat/4ad69a4c-cc9b-0dfe-0352-8b1b0cd36c7b@2ndquadrant.com
---
 doc/src/sgml/high-availability.sgml | 51 +++++++++++++++++++------
 src/backend/access/transam/xlog.c   | 59 ++++++++++++++++++++++++++---
 2 files changed, 93 insertions(+), 17 deletions(-)

diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index 19d7bd2b28..71f7f58cb0 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -2120,18 +2120,14 @@ <title>Administrator's Overview</title>
    </para>
 
    <para>
-    The setting of some parameters on the standby will need reconfiguration
-    if they have been changed on the primary. For these parameters,
-    the value on the standby must
-    be equal to or greater than the value on the primary.
-    Therefore, if you want to increase these values, you should do so on all
-    standby servers first, before applying the changes to the primary server.
-    Conversely, if you want to decrease these values, you should do so on the
-    primary server first, before applying the changes to all standby servers.
-    If these parameters
-    are not set high enough then the standby will refuse to start.
-    Higher values can then be supplied and the server
-    restarted to begin recovery again.  These parameters are:
+    The settings of some parameters determine the size of shared memory for
+    tracking transaction IDs, locks, and prepared transactions.  These shared
+    memory structures must be no smaller on a standby than on the primary in
+    order to ensure that the standby does not run out of shared memory during
+    recovery.  For example, if the primary had used a prepared transaction but
+    the standby had not allocated any shared memory for tracking prepared
+    transactions, then recovery could not continue until the standby's
+    configuration is changed.  The parameters affected are:
 
       <itemizedlist>
        <listitem>
@@ -2160,6 +2156,37 @@ <title>Administrator's Overview</title>
         </para>
        </listitem>
       </itemizedlist>
+
+    The easiest way to ensure this does not become a problem is to have these
+    parameters set on the standbys to values equal to or greater than on the
+    primary.  Therefore, if you want to increase these values, you should do
+    so on all standby servers first, before applying the changes to the
+    primary server.  Conversely, if you want to decrease these values, you
+    should do so on the primary server first, before applying the changes to
+    all standby servers.  Keep in mind that when a standby is promoted, it
+    becomes the new reference for the required parameter settings for the
+    standbys that follow it.  Therefore, to avoid this becoming a problem
+    during a switchover or failover, it is recommended to keep these settings
+    the same on all standby servers.
+   </para>
+
+   <para>
+    The WAL tracks changes to these parameters on the
+    primary.  If a hot standby processes WAL that indicates that the current
+    value on the primary is higher than its own value, it will log a warning
+    and pause recovery, for example:
+<screen>
+WARNING:  hot standby is not possible because of insufficient parameter settings
+DETAIL:  max_connections = 80 is a lower setting than on the primary server, where its value was 100.
+LOG:  recovery has paused
+DETAIL:  If recovery is unpaused, the server will shut down.
+HINT:  You can then restart the server after making the necessary configuration changes.
+</screen>
+    At that point, the settings on the standby need to be updated and the
+    instance restarted before recovery can continue.  If the standby is not a
+    hot standby, then when it encounters the incompatible parameter change, it
+    will shut down immediately without pausing, since there is then no value
+    in keeping it up.
    </para>
 
    <para>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13f1d8c3dc..f9a41e0dfb 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -6232,12 +6232,61 @@ static void
 RecoveryRequiresIntParameter(const char *param_name, int currValue, int minValue)
 {
 	if (currValue < minValue)
-		ereport(ERROR,
+	{
+		if (LocalHotStandbyActive)
+		{
+			bool		warned_for_promote = false;
+
+			ereport(WARNING,
+					(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+					 errmsg("hot standby is not possible because of insufficient parameter settings"),
+					 errdetail("%s = %d is a lower setting than on the primary server, where its value was %d.",
+							   param_name,
+							   currValue,
+							   minValue)));
+
+			SetRecoveryPause(true);
+
+			ereport(LOG,
+					(errmsg("recovery has paused"),
+					 errdetail("If recovery is unpaused, the server will shut down."),
+					 errhint("You can then restart the server after making the necessary configuration changes.")));
+
+			while (RecoveryIsPaused())
+			{
+				HandleStartupProcInterrupts();
+
+				if (CheckForStandbyTrigger())
+				{
+					if (!warned_for_promote)
+						ereport(WARNING,
+								(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+								 errmsg("promotion is not possible because of insufficient parameter settings"),
+								 /* Repeat the detail from above so it's easy to find in the log. */
+								 errdetail("%s = %d is a lower setting than on the primary server, where its value was %d.",
+										   param_name,
+										   currValue,
+										   minValue),
+								 errhint("Restart the server after making the necessary configuration changes.")));
+					warned_for_promote = true;
+				}
+
+				pgstat_report_wait_start(WAIT_EVENT_RECOVERY_PAUSE);
+				pg_usleep(1000000L);	/* 1000 ms */
+				pgstat_report_wait_end();
+			}
+		}
+
+		ereport(FATAL,
 				(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
-				 errmsg("hot standby is not possible because %s = %d is a lower setting than on the primary server (its value was %d)",
-						param_name,
-						currValue,
-						minValue)));
+				 errmsg("recovery aborted because of insufficient parameter settings"),
+				 /* Repeat the detail from above so it's easy to find in the log. */
+				 errdetail("%s = %d is a lower setting than on the primary server, where its value was %d.",
+						   param_name,
+						   currValue,
+						   minValue),
+				 errhint("You can restart the server after making the necessary configuration changes.")));
+	}
 }
 
 /*
-- 
2.29.2