Removing more vacuumlazy.c special cases, relfrozenxid optimizations

From 2cc761a55b6f727b44a32b03e8393ffd3f61fb2c Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v2] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

This new relfrozenxid optimization might not be all that valuable on its
own, but it may still facilitate future work that makes non-aggressive
VACUUMs more conscious of the benefit of advancing relfrozenxid sooner
rather than later.  In general it would be useful for non-aggressive
VACUUMs to be "more aggressive" opportunistically (e.g., by waiting for
a cleanup lock once or twice if needed).  It would also be generally
useful if aggressive VACUUMs were "less aggressive" opportunistically
(e.g. by being responsive to query cancellations when the risk of
wraparound failure is still very low).

We now also collect LP_DEAD items in the dead_tuples array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

Note that we no longer report on "pin skipped pages" in VACUUM VERBOSE,
since there is no barely any real practical sense in which we actually
miss doing useful work for these pages.  Besides, this information
always seemed to have little practical value, even to Postgres hackers.
---
 src/backend/access/heap/vacuumlazy.c          | 792 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 500 insertions(+), 301 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 282b44f87..39a7fb39e 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -284,6 +284,8 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -308,6 +310,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -322,10 +326,8 @@ typedef struct LVRelState
 	 */
 	LVDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -338,6 +340,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -377,19 +380,22 @@ static int	elevel = -1;
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void do_parallel_lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void do_parallel_lazy_cleanup_all_indexes(LVRelState *vacrel);
@@ -465,16 +471,14 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -529,6 +533,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->rel = rel;
 	vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
 					 &vacrel->indrels);
+	vacrel->aggressive = aggressive;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 
@@ -573,6 +578,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -609,30 +616,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, params);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		/*
@@ -663,28 +656,43 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM (which is the same thing as anti-wraparound
+	 * autovacuum for most practical purposes) exists so that we'll reliably
+	 * advance relfrozenxid and relminmxid sooner or later.  But we can often
+	 * opportunistically advance them even in a non-aggressive VACUUM.
+	 * Consider if that's possible now.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_prune, from before a possible relation
+	 * truncation took place. (vacrel->rel_pages is now new_rel_pages.)
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -713,7 +721,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -760,10 +767,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -771,7 +777,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->pages_removed;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -888,9 +893,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, VacuumParams *params)
 {
 	LVDeadItems *dead_items;
+	bool		aggressive;
 	BlockNumber nblocks,
 				blkno,
 				next_unskippable_block,
@@ -910,26 +916,14 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	pg_rusage_init(&ru0);
 
-	if (aggressive)
-		ereport(elevel,
-				(errmsg("aggressively vacuuming \"%s.%s\"",
-						vacrel->relnamespace,
-						vacrel->relname)));
-	else
-		ereport(elevel,
-				(errmsg("vacuuming \"%s.%s\"",
-						vacrel->relnamespace,
-						vacrel->relname)));
-
+	aggressive = vacrel->aggressive;
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
 	next_unskippable_block = 0;
 	next_failsafe_block = 0;
 	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -947,6 +941,17 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	vacrel->indstats = (IndexBulkDeleteResult **)
 		palloc0(vacrel->nindexes * sizeof(IndexBulkDeleteResult *));
 
+	if (aggressive)
+		ereport(elevel,
+				(errmsg("aggressively vacuuming \"%s.%s\"",
+						vacrel->relnamespace,
+						vacrel->relname)));
+	else
+		ereport(elevel,
+				(errmsg("vacuuming \"%s.%s\"",
+						vacrel->relnamespace,
+						vacrel->relname)));
+
 	/*
 	 * Do failsafe precheck before calling dead_items_alloc.  This ensures
 	 * that parallel VACUUM won't be attempted when relfrozenxid is already
@@ -1002,15 +1007,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
 	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
 	{
@@ -1048,18 +1044,14 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
 								 blkno, InvalidOffsetNumber);
 
+		/*
+		 * Consider need to skip blocks
+		 */
 		if (blkno == next_unskippable_block)
 		{
 			/* Time to advance next_unskippable_block */
@@ -1108,13 +1100,19 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current block can be skipped if we've seen a long enough
+			 * run of skippable blocks to justify skipping it.
+			 *
+			 * There is an exception: we will scan the table's last page to
+			 * determine whether it has tuples or not, even if it would
+			 * otherwise be skipped (unless it's clearly not worth trying to
+			 * truncate the table).  This avoids having lazy_truncate_heap()
+			 * take access-exclusive lock on the table to attempt a truncation
+			 * that just fails immediately because there are tuples in the
+			 * last page.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks &&
+				!(blkno == nblocks - 1 && should_attempt_truncation(vacrel)))
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -1124,12 +1122,22 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
 				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * in this case an approximate answer is still correct.
+				 *
+				 * (We really don't want to miss out on the opportunity to
+				 * advance relfrozenxid in a non-aggressive vacuum, but this
+				 * edge case shouldn't make that appreciably less likely in
+				 * practice.)
 				 */
 				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise, the page must be at least all-visible if not
+			 * all-frozen, so we can set all_visible_according_to_vm = true
+			 */
 			all_visible_according_to_vm = true;
 		}
 
@@ -1154,7 +1162,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		 * Consider if we definitely have enough space to process TIDs on page
 		 * already.  If we are close to overrunning the available space for
 		 * dead_items TIDs, pause and do a cycle of vacuuming before we tackle
-		 * this page.
+		 * this page.  Must do this before calling lazy_scan_prune (or before
+		 * calling lazy_scan_noprune).
 		 */
 		Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
 		if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
@@ -1189,7 +1198,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
+		 * Set up visibility map page as needed, and pin the heap page that
+		 * we're going to scan.
 		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
@@ -1202,156 +1212,52 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
+		vacrel->scanned_pages++;
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing in lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
 			bool		hastup;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
+			LockBuffer(buf, BUFFER_LOCK_SHARE);
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
+				/* Lock and pin released for us */
+				continue;
+			}
+
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup))
+			{
+				/* No need to wait for cleanup lock for this page */
+				UnlockReleaseBuffer(buf);
+				if (hastup)
+					vacrel->nonempty_pages = blkno + 1;
 				continue;
 			}
 
 			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
+			 * lazy_scan_noprune could not do all required processing without
+			 * a cleanup lock.  Wait for a cleanup lock, and then proceed to
+			 * lazy_scan_prune to perform ordinary pruning and freezing.
 			 */
-			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
-			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
-				continue;
-			}
-			if (!aggressive)
-			{
-				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
-				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
-				continue;
-			}
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Lock and pin released for us */
 			continue;
 		}
 
@@ -1564,7 +1470,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1637,14 +1543,10 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
 					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
-	appendStringInfo(&buf, ngettext("Skipped %u page due to buffer pins, ",
-									"Skipped %u pages due to buffer pins, ",
-									vacrel->pinskipped_pages),
-					 vacrel->pinskipped_pages);
-	appendStringInfo(&buf, ngettext("%u frozen page.\n",
-									"%u frozen pages.\n",
-									vacrel->frozenskipped_pages),
-					 vacrel->frozenskipped_pages);
+	appendStringInfo(&buf, ngettext("%u page skipped using visibility map.\n",
+									"%u pages skipped using visibility map.\n",
+									vacrel->rel_pages - vacrel->scanned_pages),
+					 vacrel->rel_pages - vacrel->scanned_pages);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1658,6 +1560,132 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pfree(buf.data);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a rare corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller can either hold a buffer cleanup lock on the buffer, or a simple
+ * shared lock.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never not discover the space on a promoted
+		 * standby. The harm of repeated checking ought to normally not be too
+		 * bad - the space usually should be used at some point, otherwise
+		 * there wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1764,10 +1792,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -2054,6 +2081,236 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() variant without pruning
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED in lazy_vacuum_heap_page.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).
+ *
+ * We'll always return true for a non-aggressive VACUUM, even when we know
+ * that this will cause them to miss out on freezing tuples from before
+ * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
+ * lock.  This does mean that they definitely won't be able to advance
+ * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
+ * relminmxid).
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup)
+{
+	OffsetNumber offnum,
+				maxoff;
+	bool		has_tuple_needs_freeze = false;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	*hastup = false;			/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;		/* page won't be truncatable */
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (!has_tuple_needs_freeze &&
+			heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			has_tuple_needs_freeze = true;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * We count DEAD and RECENTLY_DEAD tuples in new_dead_tuples.
+				 *
+				 * lazy_scan_prune only does this for RECENTLY_DEAD tuples,
+				 * and never has to deal with DEAD tuples directly (they
+				 * reliably become LP_DEAD items through pruning).  Our
+				 * approach to DEAD tuples is a bit arbitrary, but it seems
+				 * better than totally ignoring them.
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	if (has_tuple_needs_freeze)
+	{
+		/*
+		 * Current non-aggressive VACUUM operation definitely won't be able to
+		 * advance relfrozenxid or relminmxid
+		 */
+		Assert(!vacrel->aggressive);
+		vacrel->freeze_cutoffs_valid = false;
+	}
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in the dead_items
+	 * array iff VACUUM uses two-pass strategy case
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/*
+		 * We are not prepared to handle the corner case where a single pass
+		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
+		 * items.  Repeat the same trick that we use for DEAD tuples: pretend
+		 * that they're RECENTLY_DEAD tuples.
+		 *
+		 * There is no fundamental reason why we must take the easy way out
+		 * like this.  Finding a way to make these LP_DEAD items get set to
+		 * LP_UNUSED would be less valuable and more complicated than it is in
+		 * the two-pass strategy case, since it would necessitate that we
+		 * repeat our lazy_scan_heap caller's page-at-a-time/one-pass-strategy
+		 * heap vacuuming steps.  Whereas in the two-pass strategy case,
+		 * lazy_vacuum_heap_rel will set the LP_DEAD items to LP_UNUSED. It
+		 * must always deal with things like remaining DEAD tuples with
+		 * storage, new LP_DEAD items that we didn't see earlier on, etc.
+		 */
+		if (lpdead_items > 0)
+			*hastup = true;		/* page won't be truncatable */
+		num_tuples += lpdead_items;
+		new_dead_tuples += lpdead_items;
+	}
+	else if (lpdead_items > 0)
+	{
+		LVDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * We opt to skip FSM processing for the page on the grounds that it
+		 * is probably being modified by concurrent DML operations.  Seems
+		 * best to assume that the space is best left behind for future
+		 * updates of existing tuples.  This matches what opportunistic
+		 * pruning does.
+		 *
+		 * It's theoretically possible for us to set VM bits here too, but we
+		 * don't try that either.  It is highly unlikely to be possible, much
+		 * less useful.
+		 */
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Remove the collected garbage tuples from the table and its indexes.
  *
@@ -2500,67 +2757,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2655,7 +2851,7 @@ do_parallel_lazy_cleanup_all_indexes(LVRelState *vacrel)
 	 */
 	vacrel->lps->lvshared->reltuples = vacrel->new_rel_tuples;
 	vacrel->lps->lvshared->estimated_count =
-		(vacrel->tupcount_pages < vacrel->rel_pages);
+		(vacrel->scanned_pages < vacrel->rel_pages);
 
 	/* Determine the number of parallel workers to launch */
 	if (vacrel->lps->lvshared->first_time)
@@ -2972,7 +3168,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -3123,7 +3319,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From e867662d06d8db72b7be4e26f509c57029d867f9 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v3 1/2] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

We now also collect LP_DEAD items in the dead_tuples array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

We no longer report on "pin skipped pages" in log output.  A later patch
will add back an improved version of the same instrumentation.  We don't
want to show any information about any failures to acquire cleanup locks
unless we actually failed to do useful work as a consequence.  A page
that we could not acquire a cleanup lock on is now treated as equivalent
to any other scanned page in most cases.
---
 src/backend/access/heap/vacuumlazy.c          | 856 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 541 insertions(+), 324 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 282b44f87..1fb8735a2 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -151,7 +151,7 @@ typedef enum
 /*
  * LVDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
  * Each TID points to an LP_DEAD line pointer from a heap page that has been
- * processed by lazy_scan_prune.
+ * processed by lazy_scan_prune (or by lazy_scan_noprune, perhaps).
  *
  * Also needed by lazy_vacuum_heap_rel, which marks the same LP_DEAD line
  * pointers as LP_UNUSED during second heap pass.
@@ -284,6 +284,8 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -308,6 +310,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -322,10 +326,8 @@ typedef struct LVRelState
 	 */
 	LVDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -338,6 +340,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -377,19 +380,22 @@ static int	elevel = -1;
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup, bool *hasfreespace);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void do_parallel_lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void do_parallel_lazy_cleanup_all_indexes(LVRelState *vacrel);
@@ -465,16 +471,14 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -529,6 +533,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->rel = rel;
 	vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
 					 &vacrel->indrels);
+	vacrel->aggressive = aggressive;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 
@@ -573,6 +578,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -609,30 +616,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, params);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		/*
@@ -663,28 +656,44 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
+	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
+	 * provided we didn't skip any all-visible (not all-frozen) pages using
+	 * the visibility map, and assuming that we didn't fail to get a cleanup
+	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
+	 * MultiXactCutoff) established for VACUUM operation.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_heap, which won't match when we
+	 * happened to truncate the relation afterwards.
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -713,7 +722,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -760,10 +768,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -771,7 +778,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->pages_removed;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -888,7 +894,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, VacuumParams *params)
 {
 	LVDeadItems *dead_items;
 	BlockNumber nblocks,
@@ -910,7 +916,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	pg_rusage_init(&ru0);
 
-	if (aggressive)
+	if (vacrel->aggressive)
 		ereport(elevel,
 				(errmsg("aggressively vacuuming \"%s.%s\"",
 						vacrel->relnamespace,
@@ -922,14 +928,11 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 						vacrel->relname)));
 
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
-	next_unskippable_block = 0;
 	next_failsafe_block = 0;
 	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -969,7 +972,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/*
-	 * Except when aggressive is set, we want to skip pages that are
+	 * Set things up for skipping blocks using visibility map.
+	 *
+	 * Except when vacrel->aggressive is set, we want to skip pages that are
 	 * all-visible according to the visibility map, but only when we can skip
 	 * at least SKIP_PAGES_THRESHOLD consecutive pages.  Since we're reading
 	 * sequentially, the OS should be doing readahead for us, so there's no
@@ -978,8 +983,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * page means that we can't update relfrozenxid, so we only want to do it
 	 * if we can skip a goodly number of pages.
 	 *
-	 * When aggressive is set, we can't skip pages just because they are
-	 * all-visible, but we can still skip pages that are all-frozen, since
+	 * When vacrel->aggressive is set, we can't skip pages just because they
+	 * are all-visible, but we can still skip pages that are all-frozen, since
 	 * such pages do not need freezing and do not affect the value that we can
 	 * safely set for relfrozenxid or relminmxid.
 	 *
@@ -1002,18 +1007,11 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
+	skipping_blocks = false;
 	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
 	{
+		next_unskippable_block = 0;
 		while (next_unskippable_block < nblocks)
 		{
 			uint8		vmstatus;
@@ -1021,7 +1019,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vmstatus = visibilitymap_get_status(vacrel->rel,
 												next_unskippable_block,
 												&vmbuffer);
-			if (aggressive)
+			if (vacrel->aggressive)
 			{
 				if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
 					break;
@@ -1034,12 +1032,11 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vacuum_delay_point();
 			next_unskippable_block++;
 		}
+		if (next_unskippable_block >= SKIP_PAGES_THRESHOLD)
+			skipping_blocks = true;
 	}
-
-	if (next_unskippable_block >= SKIP_PAGES_THRESHOLD)
-		skipping_blocks = true;
 	else
-		skipping_blocks = false;
+		next_unskippable_block = InvalidBlockNumber;
 
 	for (blkno = 0; blkno < nblocks; blkno++)
 	{
@@ -1048,44 +1045,38 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
 								 blkno, InvalidOffsetNumber);
 
+		/*
+		 * Consider need to skip blocks using visibility map
+		 */
 		if (blkno == next_unskippable_block)
 		{
 			/* Time to advance next_unskippable_block */
+			Assert((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0);
 			next_unskippable_block++;
-			if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+			while (next_unskippable_block < nblocks)
 			{
-				while (next_unskippable_block < nblocks)
-				{
-					uint8		vmskipflags;
+				uint8		vmskipflags;
 
-					vmskipflags = visibilitymap_get_status(vacrel->rel,
-														   next_unskippable_block,
-														   &vmbuffer);
-					if (aggressive)
-					{
-						if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
-							break;
-					}
-					else
-					{
-						if ((vmskipflags & VISIBILITYMAP_ALL_VISIBLE) == 0)
-							break;
-					}
-					vacuum_delay_point();
-					next_unskippable_block++;
+				vmskipflags = visibilitymap_get_status(vacrel->rel,
+													   next_unskippable_block,
+													   &vmbuffer);
+				if (vacrel->aggressive)
+				{
+					if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
+						break;
 				}
+				else
+				{
+					if ((vmskipflags & VISIBILITYMAP_ALL_VISIBLE) == 0)
+						break;
+				}
+				vacuum_delay_point();
+				next_unskippable_block++;
 			}
 
 			/*
@@ -1102,19 +1093,24 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			 * it's not all-visible.  But in an aggressive vacuum we know only
 			 * that it's not all-frozen, so it might still be all-visible.
 			 */
-			if (aggressive && VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
+			if (vacrel->aggressive &&
+				VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
 				all_visible_according_to_vm = true;
 		}
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current page can be skipped if we've seen a long enough run
+			 * of skippable blocks to justify skipping it -- provided it's not
+			 * the last page in the relation (according to rel_pages/nblocks).
+			 *
+			 * We always scan the table's last page to determine whether it
+			 * has tuples or not, even if it would otherwise be skipped.  This
+			 * avoids having lazy_truncate_heap() take access-exclusive lock
+			 * on the table to attempt a truncation that just fails
+			 * immediately because there are tuples on the last page.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks && blkno < nblocks - 1)
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -1123,18 +1119,31 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * careful to count it as a skipped all-frozen page in that
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
-				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * know whether it was initially all-frozen, so we have to
+				 * recheck.
 				 */
-				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+				if (vacrel->aggressive ||
+					VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise we scan the page.  It must be at least all-visible,
+			 * if not all-frozen.
+			 */
 			all_visible_according_to_vm = true;
 		}
 
 		vacuum_delay_point();
 
+		/*
+		 * We're not skipping this page using the visibility map, and so it is
+		 * (by definition) a scanned page.  Any tuples from this page are now
+		 * guaranteed to be counted below, after some preparatory checks.
+		 */
+		vacrel->scanned_pages++;
+
 		/*
 		 * Regularly check if wraparound failsafe should trigger.
 		 *
@@ -1189,174 +1198,78 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
-		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
-		 * already have the correct page pinned anyway.  However, it's
-		 * possible that (a) next_unskippable_block is covered by a different
-		 * VM page than the current block or (b) we released our pin and did a
-		 * cycle of index vacuuming.
+		 * already have the correct page pinned anyway.
 		 */
 		visibilitymap_pin(vacrel->rel, blkno, &vmbuffer);
 
+		/* Finished preparatory checks.  Actually scan the page. */
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing using lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
-			bool		hastup;
+			bool		hastup,
+						hasfreespace;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
-			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				continue;
-			}
-
-			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
-			 */
 			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
+				/* Processed as new/empty page (lock and pin released) */
 				continue;
 			}
-			if (!aggressive)
+
+			/* Collect LP_DEAD items in dead_items array, count tuples */
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
+								  &hasfreespace))
 			{
+				Size		freespace;
+
 				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
+				 * Processed page successfully (without cleanup lock) -- just
+				 * need to perform rel truncation and FSM steps, much like the
+				 * lazy_scan_prune case.  Don't bother trying to match its
+				 * visibility map setting steps, though.
 				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
 				if (hastup)
 					vacrel->nonempty_pages = blkno + 1;
+				if (hasfreespace)
+					freespace = PageGetHeapFreeSpace(page);
+				UnlockReleaseBuffer(buf);
+				if (hasfreespace)
+					RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
 			}
+
+			/*
+			 * lazy_scan_noprune could not do all required processing.  Wait
+			 * for a cleanup lock, and call lazy_scan_prune in the usual way.
+			 */
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Processed as new/empty page (lock and pin released) */
 			continue;
 		}
 
 		/*
-		 * Prune and freeze tuples.
+		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
 		 * dead_items array.  This includes LP_DEAD line pointers that we
@@ -1564,7 +1477,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1637,14 +1550,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
 					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
-	appendStringInfo(&buf, ngettext("Skipped %u page due to buffer pins, ",
-									"Skipped %u pages due to buffer pins, ",
-									vacrel->pinskipped_pages),
-					 vacrel->pinskipped_pages);
-	appendStringInfo(&buf, ngettext("%u frozen page.\n",
-									"%u frozen pages.\n",
-									vacrel->frozenskipped_pages),
-					 vacrel->frozenskipped_pages);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1658,6 +1563,138 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pfree(buf.data);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller must hold at least a shared lock.  We might need to escalate the
+ * lock in that case, so the type of lock caller holds needs to be specified
+ * using 'sharelock' argument.
+ *
+ * Returns false in common case where caller should go on to call
+ * lazy_scan_prune (or lazy_scan_noprune).  Otherwise returns true, indicating
+ * that lazy_scan_heap is done processing the page, releasing lock on caller's
+ * behalf.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never discover the space on a promoted standby.
+		 * The harm of repeated checking ought to normally not be too bad.
+		 * The space usually should be used at some point, otherwise there
+		 * wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1702,6 +1739,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
 	maxoff = PageGetMaxOffsetNumber(page);
 
 retry:
@@ -1764,10 +1803,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -2054,6 +2092,243 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() variant without pruning
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED in lazy_vacuum_heap_page.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).
+ *
+ * Returns true to indicate that all required processing has been performed.
+ * We'll always return true for a non-aggressive VACUUM, even when we know
+ * that this will cause them to miss out on freezing tuples from before
+ * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
+ * lock.  This does mean that they definitely won't be able to advance
+ * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
+ * relminmxid).  Caller waits for full cleanup lock when we return false.
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.  The
+ * hasfreespace flag instructs caller on whether or not it should do generic
+ * FSM processing for page, which is determined based on almost the same
+ * criteria as the lazy_scan_prune case.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup,
+				  bool *hasfreespace)
+{
+	OffsetNumber offnum,
+				maxoff;
+	bool		has_tuple_needs_freeze = false;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
+	*hastup = false;			/* for now */
+	*hasfreespace = false;		/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;		/* page prevents rel truncation */
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (!has_tuple_needs_freeze &&
+			heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			has_tuple_needs_freeze = true;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+
+				/*
+				 * There is some useful work for pruning to do, that won't be
+				 * done due to failure to get a cleanup lock.
+				 *
+				 * TODO Add dedicated instrumentation for this case
+				 */
+				break;
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	if (has_tuple_needs_freeze)
+	{
+		/*
+		 * Current non-aggressive VACUUM operation definitely won't be able to
+		 * advance relfrozenxid or relminmxid
+		 */
+		Assert(!vacrel->aggressive);
+		vacrel->freeze_cutoffs_valid = false;
+	}
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in the dead_items
+	 * array iff VACUUM uses two-pass strategy case
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/*
+		 * We are not prepared to handle the corner case where a single pass
+		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
+		 * items.  Count the LP_DEAD items as if they were DEAD tuples with
+		 * storage that we cannot prune away.  This is slightly inaccurate,
+		 * but it hardly seems worth having dedicated handling just for this
+		 * case.
+		 *
+		 * There is no fundamental reason why we must take the easy way out
+		 * like this.  Finding a way to make these LP_DEAD items get set to
+		 * LP_UNUSED would be less valuable and more complicated than it is in
+		 * the two-pass strategy case, since it would necessitate that we
+		 * repeat our lazy_scan_heap caller's page-at-a-time/one-pass-strategy
+		 * heap vacuuming steps.  Whereas in the two-pass strategy case,
+		 * lazy_vacuum_heap_rel will set the LP_DEAD items to LP_UNUSED. It
+		 * must always deal with things like remaining DEAD tuples with
+		 * storage, new LP_DEAD items that we didn't see earlier on, etc.
+		 */
+		if (lpdead_items > 0)
+			*hastup = true;
+		*hasfreespace = true;
+		num_tuples += lpdead_items;
+		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+	}
+	else if (lpdead_items > 0)
+	{
+		LVDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * Caller won't be vacuuming this page later, so tell it to record
+		 * page's freespace in the FSM now
+		 */
+		*hasfreespace = true;
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Remove the collected garbage tuples from the table and its indexes.
  *
@@ -2500,67 +2775,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2655,7 +2869,7 @@ do_parallel_lazy_cleanup_all_indexes(LVRelState *vacrel)
 	 */
 	vacrel->lps->lvshared->reltuples = vacrel->new_rel_tuples;
 	vacrel->lps->lvshared->estimated_count =
-		(vacrel->tupcount_pages < vacrel->rel_pages);
+		(vacrel->scanned_pages < vacrel->rel_pages);
 
 	/* Determine the number of parallel workers to launch */
 	if (vacrel->lps->lvshared->first_time)
@@ -2972,7 +3186,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -3123,7 +3337,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From 4affd72e80405f6593866c21391789a5afe471f0 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 21 Nov 2021 14:47:11 -0800
Subject: [PATCH v3 2/2] Improve log_autovacuum_min_duration output.

Add instrumentation of "missed dead tuples", and the number of pages
that had at least one such tuple.  These are fully DEAD (not just
RECENTLY_DEAD) tuples with storage that could not be pruned due to an
inability to acquire a cleanup lock.  This is a replacement for the
"skipped due to pin" instrumentation removed by the previous commit.
Note that the new instrumentation doesn't say anything about pages that
we failed to acquire a cleanup lock on when we see that there were no
missed dead tuples on the page.

Also report on visibility map pages skipped by VACUUM, without regard
for whether the pages were all-frozen or just all-visible.

Also report when and how relfrozenxid is advanced by VACUUM, including
non-aggressive VACUUM.  Apart from being useful on its own, this might
enable future work that teaches non-aggressive VACUUM to be more
concerned about advancing relfrozenxid sooner rather than later.

Also enhance how we report OldestXmin cutoff by putting it in context:
show how far behind it is at the _end_ of the VACUUM operation.

Deliberately don't do anything with VACUUM VERBOSE in this commit, since
a pending patch will generalize the log_autovacuum_min_duration code to
produce VACUUM VERBOSE output as well [1].  That'll get committed first.

[1] https://commitfest.postgresql.org/36/3431/
---
 src/include/commands/vacuum.h        |   2 +
 src/backend/access/heap/vacuumlazy.c | 100 +++++++++++++++++++--------
 src/backend/commands/analyze.c       |   3 +
 src/backend/commands/vacuum.c        |   9 +++
 4 files changed, 84 insertions(+), 30 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 4cfd52eaf..bc625463e 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -263,6 +263,8 @@ extern void vac_update_relstats(Relation relation,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
+								bool *frozenxid_updated,
+								bool *minmulti_updated,
 								bool in_outer_xact);
 extern void vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1fb8735a2..56da89f04 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -330,6 +330,7 @@ typedef struct LVRelState
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
+	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
 
 	/* Statistics output by us, for table */
@@ -343,8 +344,8 @@ typedef struct LVRelState
 	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
-	int64		new_dead_tuples;	/* new estimated total # of dead items in
-									 * table */
+	int64		recently_dead_tuples;	/* # dead, but not yet removable */
+	int64		missed_dead_tuples;	/* # removable, but not removed */
 	int64		num_tuples;		/* total number of nonremovable tuples */
 	int64		live_tuples;	/* live tuples (reltuples estimate) */
 } LVRelState;
@@ -472,6 +473,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	double		read_rate,
 				write_rate;
 	bool		aggressive;
+	bool		frozenxid_updated,
+				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
@@ -681,9 +684,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
+		frozenxid_updated = minmulti_updated = false;
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							InvalidTransactionId, InvalidMultiXactId, false);
+							InvalidTransactionId, InvalidMultiXactId,
+							NULL, NULL, false);
 	}
 	else
 	{
@@ -692,7 +697,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff, false);
+							FreezeLimit, MultiXactCutoff,
+							&frozenxid_updated, &minmulti_updated, false);
 	}
 
 	/*
@@ -708,7 +714,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_report_vacuum(RelationGetRelid(rel),
 						 rel->rd_rel->relisshared,
 						 Max(new_live_tuples, 0),
-						 vacrel->new_dead_tuples);
+						 vacrel->recently_dead_tuples +
+						 vacrel->missed_dead_tuples);
 	pgstat_progress_end_command();
 
 	/* and log the action if appropriate */
@@ -722,6 +729,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
+			int32		   diff;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -768,16 +776,40 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped using visibility map (%.2f%% of total)\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->frozenskipped_pages);
+							 orig_rel_pages - vacrel->scanned_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * (orig_rel_pages - vacrel->scanned_pages) / orig_rel_pages);
 			appendStringInfo(&buf,
-							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
+							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
 							 (long long) vacrel->new_rel_tuples,
-							 (long long) vacrel->new_dead_tuples,
-							 OldestXmin);
+							 (long long) vacrel->recently_dead_tuples);
+			if (vacrel->missed_dead_tuples > 0)
+				appendStringInfo(&buf,
+								 _("tuples missed: %lld dead from %u contended pages\n"),
+								 (long long) vacrel->missed_dead_tuples,
+								 vacrel->missed_dead_pages);
+			diff = (int32) (ReadNextTransactionId() - OldestXmin);
+			appendStringInfo(&buf,
+							 _("removal cutoff: oldest xmin was %u, which is now %d xact IDs behind\n"),
+							 OldestXmin, diff);
+			if (frozenxid_updated)
+			{
+				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				appendStringInfo(&buf,
+								 _("relfrozenxid: advanced by %d xact IDs, new value: %u\n"),
+								 diff, FreezeLimit);
+			}
+			if (minmulti_updated)
+			{
+				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				appendStringInfo(&buf,
+								 _("relminmxid: advanced by %d multixact IDs, new value: %u\n"),
+								 diff, MultiXactCutoff);
+			}
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -935,13 +967,15 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params)
 	vacrel->frozenskipped_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
+	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
 
 	/* Initialize instrumentation counters */
 	vacrel->num_index_scans = 0;
 	vacrel->tuples_deleted = 0;
 	vacrel->lpdead_items = 0;
-	vacrel->new_dead_tuples = 0;
+	vacrel->recently_dead_tuples = 0;
+	vacrel->missed_dead_tuples = 0;
 	vacrel->num_tuples = 0;
 	vacrel->live_tuples = 0;
 
@@ -1485,7 +1519,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params)
 	 * (unlikely) scenario that new_live_tuples is -1, take it as zero.
 	 */
 	vacrel->new_rel_tuples =
-		Max(vacrel->new_live_tuples, 0) + vacrel->new_dead_tuples;
+		Max(vacrel->new_live_tuples, 0) + vacrel->recently_dead_tuples +
+		vacrel->missed_dead_tuples;
 
 	/*
 	 * Release any remaining pin on visibility map page.
@@ -1549,7 +1584,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params)
 	initStringInfo(&buf);
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
-					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
+					 (long long) vacrel->recently_dead_tuples,
+					 vacrel->OldestXmin);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1731,7 +1767,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				lpdead_items,
-				new_dead_tuples,
+				recently_dead_tuples,
 				num_tuples,
 				live_tuples;
 	int			nnewlpdead;
@@ -1748,7 +1784,7 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	tuples_deleted = 0;
 	lpdead_items = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
 	num_tuples = 0;
 	live_tuples = 0;
 
@@ -1907,11 +1943,11 @@ retry:
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * If tuple is recently deleted then we must not remove it
-				 * from relation.  (We only remove items that are LP_DEAD from
+				 * If tuple is recently dead then we must not remove it from
+				 * the relation.  (We only remove items that are LP_DEAD from
 				 * pruning.)
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				prunestate->all_visible = false;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -2087,7 +2123,7 @@ retry:
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
 	vacrel->lpdead_items += lpdead_items;
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
 }
@@ -2141,7 +2177,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	int			lpdead_items,
 				num_tuples,
 				live_tuples,
-				new_dead_tuples;
+				recently_dead_tuples,
+				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 
@@ -2153,7 +2190,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	lpdead_items = 0;
 	num_tuples = 0;
 	live_tuples = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
+	missed_dead_tuples = 0;
 
 	maxoff = PageGetMaxOffsetNumber(page);
 	for (offnum = FirstOffsetNumber;
@@ -2222,16 +2260,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 				/*
 				 * There is some useful work for pruning to do, that won't be
 				 * done due to failure to get a cleanup lock.
-				 *
-				 * TODO Add dedicated instrumentation for this case
 				 */
+				missed_dead_tuples++;
 				break;
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 * Count in recently_dead_tuples, just like lazy_scan_prune
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
 
@@ -2286,7 +2323,7 @@ lazy_scan_noprune(LVRelState *vacrel,
 			*hastup = true;
 		*hasfreespace = true;
 		num_tuples += lpdead_items;
-		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+		missed_dead_tuples += lpdead_items;
 	}
 	else if (lpdead_items > 0)
 	{
@@ -2321,9 +2358,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 	/*
 	 * Finally, add relevant page-local counts to whole-VACUUM counts
 	 */
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
+	vacrel->missed_dead_tuples += missed_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
+	if (missed_dead_tuples > 0)
+		vacrel->missed_dead_pages++;
 
 	/* Caller won't need to call lazy_scan_prune with same page */
 	return true;
@@ -2397,8 +2437,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * dead_items space is not CPU cache resident.
 		 *
 		 * We don't take any special steps to remember the LP_DEAD items (such
-		 * as counting them in new_dead_tuples report to the stats collector)
-		 * when the optimization is applied.  Though the accounting used in
+		 * as counting them in our final report to the stats collector) when
+		 * the optimization is applied.  Though the accounting used in
 		 * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD
 		 * items as dead rows in its own stats collector report, that's okay.
 		 * The discrepancy should be negligible.  If this optimization is ever
@@ -4058,7 +4098,7 @@ update_index_statistics(LVRelState *vacrel)
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
-							false);
+							NULL, NULL, false);
 	}
 }
 
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd77907fc..afd1cb8f5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -651,6 +651,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 
 		/* Same for indexes */
@@ -667,6 +668,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
+								NULL, NULL,
 								in_outer_xact);
 		}
 	}
@@ -679,6 +681,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		vac_update_relstats(onerel, -1, totalrows,
 							0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 	}
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 5c4bc15b4..8bd4bd12c 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1308,6 +1308,7 @@ vac_update_relstats(Relation relation,
 					BlockNumber num_all_visible_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
+					bool *frozenxid_updated, bool *minmulti_updated,
 					bool in_outer_xact)
 {
 	Oid			relid = RelationGetRelid(relation);
@@ -1383,22 +1384,30 @@ vac_update_relstats(Relation relation,
 	 * This should match vac_update_datfrozenxid() concerning what we consider
 	 * to be "in the future".
 	 */
+	if (frozenxid_updated)
+		*frozenxid_updated = false;
 	if (TransactionIdIsNormal(frozenxid) &&
 		pgcform->relfrozenxid != frozenxid &&
 		(TransactionIdPrecedes(pgcform->relfrozenxid, frozenxid) ||
 		 TransactionIdPrecedes(ReadNextTransactionId(),
 							   pgcform->relfrozenxid)))
 	{
+		if (frozenxid_updated)
+			*frozenxid_updated = true;
 		pgcform->relfrozenxid = frozenxid;
 		dirty = true;
 	}
 
 	/* Similarly for relminmxid */
+	if (minmulti_updated)
+		*minmulti_updated = false;
 	if (MultiXactIdIsValid(minmulti) &&
 		pgcform->relminmxid != minmulti &&
 		(MultiXactIdPrecedes(pgcform->relminmxid, minmulti) ||
 		 MultiXactIdPrecedes(ReadNextMultiXactId(), pgcform->relminmxid)))
 	{
+		if (minmulti_updated)
+			*minmulti_updated = true;
 		pgcform->relminmxid = minmulti;
 		dirty = true;
 	}
-- 
2.30.2

From 276122392aefcf0d7a079d7c7dc532228d495bd7 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 21 Nov 2021 14:47:11 -0800
Subject: [PATCH v4 2/5] Improve log_autovacuum_min_duration output.

Add instrumentation of "missed dead tuples", and the number of pages
that had at least one such tuple.  These are fully DEAD (not just
RECENTLY_DEAD) tuples with storage that could not be pruned due to an
inability to acquire a cleanup lock.  This is a replacement for the
"skipped due to pin" instrumentation removed by the previous commit.
Note that the new instrumentation doesn't say anything about pages that
we failed to acquire a cleanup lock on when we see that there were no
missed dead tuples on the page.

Also report on visibility map pages skipped by VACUUM, without regard
for whether the pages were all-frozen or just all-visible.

Also report when and how relfrozenxid is advanced by VACUUM, including
non-aggressive VACUUM.  Apart from being useful on its own, this might
enable future work that teaches non-aggressive VACUUM to be more
concerned about advancing relfrozenxid sooner rather than later.

Also report number of tuples frozen.  This will become more important
when the later patch to perform opportunistic tuple freezing is
committed.

Also enhance how we report OldestXmin cutoff by putting it in context:
show how far behind it is at the _end_ of the VACUUM operation.

Deliberately don't do anything with VACUUM VERBOSE in this commit, since
a pending patch will generalize the log_autovacuum_min_duration code to
produce VACUUM VERBOSE output as well [1].  That'll get committed first.

[1] https://commitfest.postgresql.org/36/3431/
---
 src/include/commands/vacuum.h        |   2 +
 src/backend/access/heap/vacuumlazy.c | 108 +++++++++++++++++++--------
 src/backend/commands/analyze.c       |   3 +
 src/backend/commands/vacuum.c        |   9 +++
 4 files changed, 91 insertions(+), 31 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 4cfd52eaf..bc625463e 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -263,6 +263,8 @@ extern void vac_update_relstats(Relation relation,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
+								bool *frozenxid_updated,
+								bool *minmulti_updated,
 								bool in_outer_xact);
 extern void vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index c6d3a483f..238e07a78 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -351,6 +351,7 @@ typedef struct LVRelState
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
+	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
 
 	/* Statistics output by us, for table */
@@ -363,9 +364,10 @@ typedef struct LVRelState
 	int			num_index_scans;
 	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
+	int64		tuples_frozen;	/* # frozen by us */
 	int64		lpdead_items;	/* # deleted from indexes */
-	int64		new_dead_tuples;	/* new estimated total # of dead items in
-									 * table */
+	int64		recently_dead_tuples;	/* # dead, but not yet removable */
+	int64		missed_dead_tuples; /* # removable, but not removed */
 	int64		num_tuples;		/* total number of nonremovable tuples */
 	int64		live_tuples;	/* live tuples (reltuples estimate) */
 } LVRelState;
@@ -488,6 +490,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				write_rate;
 	bool		aggressive,
 				skipwithvm;
+	bool		frozenxid_updated,
+				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
@@ -705,9 +709,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
+		frozenxid_updated = minmulti_updated = false;
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							InvalidTransactionId, InvalidMultiXactId, false);
+							InvalidTransactionId, InvalidMultiXactId,
+							NULL, NULL, false);
 	}
 	else
 	{
@@ -716,7 +722,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff, false);
+							FreezeLimit, MultiXactCutoff,
+							&frozenxid_updated, &minmulti_updated, false);
 	}
 
 	/*
@@ -732,7 +739,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_report_vacuum(RelationGetRelid(rel),
 						 rel->rd_rel->relisshared,
 						 Max(new_live_tuples, 0),
-						 vacrel->new_dead_tuples);
+						 vacrel->recently_dead_tuples +
+						 vacrel->missed_dead_tuples);
 	pgstat_progress_end_command();
 
 	/* and log the action if appropriate */
@@ -746,6 +754,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
+			int32		diff;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -792,16 +801,41 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped using visibility map (%.2f%% of total)\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->frozenskipped_pages);
+							 orig_rel_pages - vacrel->scanned_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * (orig_rel_pages - vacrel->scanned_pages) / orig_rel_pages);
 			appendStringInfo(&buf,
-							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
+							 _("tuples: %lld removed, %lld remain (%lld newly frozen), %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
 							 (long long) vacrel->new_rel_tuples,
-							 (long long) vacrel->new_dead_tuples,
-							 OldestXmin);
+							 (long long) vacrel->tuples_frozen,
+							 (long long) vacrel->recently_dead_tuples);
+			if (vacrel->missed_dead_tuples > 0)
+				appendStringInfo(&buf,
+								 _("tuples missed: %lld dead from %u contended pages\n"),
+								 (long long) vacrel->missed_dead_tuples,
+								 vacrel->missed_dead_pages);
+			diff = (int32) (ReadNextTransactionId() - OldestXmin);
+			appendStringInfo(&buf,
+							 _("removal cutoff: oldest xmin was %u, which is now %d xact IDs behind\n"),
+							 OldestXmin, diff);
+			if (frozenxid_updated)
+			{
+				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				appendStringInfo(&buf,
+								 _("relfrozenxid: advanced by %d xact IDs, new value: %u\n"),
+								 diff, FreezeLimit);
+			}
+			if (minmulti_updated)
+			{
+				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				appendStringInfo(&buf,
+								 _("relminmxid: advanced by %d multixact IDs, new value: %u\n"),
+								 diff, MultiXactCutoff);
+			}
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -957,13 +991,16 @@ lazy_scan_heap(LVRelState *vacrel, bool skipwithvm, int nworkers)
 	vacrel->frozenskipped_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
+	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
 
 	/* Initialize instrumentation counters */
 	vacrel->num_index_scans = 0;
 	vacrel->tuples_deleted = 0;
+	vacrel->tuples_frozen = 0;
 	vacrel->lpdead_items = 0;
-	vacrel->new_dead_tuples = 0;
+	vacrel->recently_dead_tuples = 0;
+	vacrel->missed_dead_tuples = 0;
 	vacrel->num_tuples = 0;
 	vacrel->live_tuples = 0;
 
@@ -1510,7 +1547,8 @@ lazy_scan_heap(LVRelState *vacrel, bool skipwithvm, int nworkers)
 	 * (unlikely) scenario that new_live_tuples is -1, take it as zero.
 	 */
 	vacrel->new_rel_tuples =
-		Max(vacrel->new_live_tuples, 0) + vacrel->new_dead_tuples;
+		Max(vacrel->new_live_tuples, 0) + vacrel->recently_dead_tuples +
+		vacrel->missed_dead_tuples;
 
 	/*
 	 * Release any remaining pin on visibility map page.
@@ -1574,7 +1612,8 @@ lazy_scan_heap(LVRelState *vacrel, bool skipwithvm, int nworkers)
 	initStringInfo(&buf);
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
-					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
+					 (long long) vacrel->recently_dead_tuples,
+					 vacrel->OldestXmin);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1756,7 +1795,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				lpdead_items,
-				new_dead_tuples,
+				recently_dead_tuples,
 				num_tuples,
 				live_tuples;
 	int			nnewlpdead;
@@ -1773,7 +1812,7 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	tuples_deleted = 0;
 	lpdead_items = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
 	num_tuples = 0;
 	live_tuples = 0;
 
@@ -1932,11 +1971,11 @@ retry:
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * If tuple is recently deleted then we must not remove it
-				 * from relation.  (We only remove items that are LP_DEAD from
+				 * If tuple is recently dead then we must not remove it from
+				 * the relation.  (We only remove items that are LP_DEAD from
 				 * pruning.)
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				prunestate->all_visible = false;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -2111,8 +2150,9 @@ retry:
 
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
+	vacrel->tuples_frozen += nfrozen;
 	vacrel->lpdead_items += lpdead_items;
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
 }
@@ -2165,7 +2205,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	int			lpdead_items,
 				num_tuples,
 				live_tuples,
-				new_dead_tuples;
+				recently_dead_tuples,
+				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 
@@ -2177,7 +2218,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	lpdead_items = 0;
 	num_tuples = 0;
 	live_tuples = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
+	missed_dead_tuples = 0;
 
 	maxoff = PageGetMaxOffsetNumber(page);
 	for (offnum = FirstOffsetNumber;
@@ -2250,16 +2292,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 				/*
 				 * There is some useful work for pruning to do, that won't be
 				 * done due to failure to get a cleanup lock.
-				 *
-				 * TODO Add dedicated instrumentation for this case
 				 */
+				missed_dead_tuples++;
 				break;
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 * Count in recently_dead_tuples, just like lazy_scan_prune
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
 
@@ -2287,13 +2328,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 		 *
 		 * We are not prepared to handle the corner case where a single pass
 		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
-		 * items.
+		 * items.  Count the LP_DEAD items as missed_dead_tuples instead. This
+		 * is slightly dishonest, but it's better than maintaining code to do
+		 * heap vacuuming for this one narrow corner case.
 		 */
 		if (lpdead_items > 0)
 			*hastup = true;
 		*hasfreespace = true;
 		num_tuples += lpdead_items;
-		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+		missed_dead_tuples += lpdead_items;
 	}
 	else if (lpdead_items > 0)
 	{
@@ -2328,9 +2371,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 	/*
 	 * Finally, add relevant page-local counts to whole-VACUUM counts
 	 */
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
+	vacrel->missed_dead_tuples += missed_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
+	if (missed_dead_tuples > 0)
+		vacrel->missed_dead_pages++;
 
 	/* Caller won't need to call lazy_scan_prune with same page */
 	return true;
@@ -2404,8 +2450,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * dead_items space is not CPU cache resident.
 		 *
 		 * We don't take any special steps to remember the LP_DEAD items (such
-		 * as counting them in new_dead_tuples report to the stats collector)
-		 * when the optimization is applied.  Though the accounting used in
+		 * as counting them in our final report to the stats collector) when
+		 * the optimization is applied.  Though the accounting used in
 		 * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD
 		 * items as dead rows in its own stats collector report, that's okay.
 		 * The discrepancy should be negligible.  If this optimization is ever
@@ -4061,7 +4107,7 @@ update_index_statistics(LVRelState *vacrel)
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
-							false);
+							NULL, NULL, false);
 	}
 }
 
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd77907fc..afd1cb8f5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -651,6 +651,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 
 		/* Same for indexes */
@@ -667,6 +668,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
+								NULL, NULL,
 								in_outer_xact);
 		}
 	}
@@ -679,6 +681,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		vac_update_relstats(onerel, -1, totalrows,
 							0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 	}
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 5c4bc15b4..8bd4bd12c 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1308,6 +1308,7 @@ vac_update_relstats(Relation relation,
 					BlockNumber num_all_visible_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
+					bool *frozenxid_updated, bool *minmulti_updated,
 					bool in_outer_xact)
 {
 	Oid			relid = RelationGetRelid(relation);
@@ -1383,22 +1384,30 @@ vac_update_relstats(Relation relation,
 	 * This should match vac_update_datfrozenxid() concerning what we consider
 	 * to be "in the future".
 	 */
+	if (frozenxid_updated)
+		*frozenxid_updated = false;
 	if (TransactionIdIsNormal(frozenxid) &&
 		pgcform->relfrozenxid != frozenxid &&
 		(TransactionIdPrecedes(pgcform->relfrozenxid, frozenxid) ||
 		 TransactionIdPrecedes(ReadNextTransactionId(),
 							   pgcform->relfrozenxid)))
 	{
+		if (frozenxid_updated)
+			*frozenxid_updated = true;
 		pgcform->relfrozenxid = frozenxid;
 		dirty = true;
 	}
 
 	/* Similarly for relminmxid */
+	if (minmulti_updated)
+		*minmulti_updated = false;
 	if (MultiXactIdIsValid(minmulti) &&
 		pgcform->relminmxid != minmulti &&
 		(MultiXactIdPrecedes(pgcform->relminmxid, minmulti) ||
 		 MultiXactIdPrecedes(ReadNextMultiXactId(), pgcform->relminmxid)))
 	{
+		if (minmulti_updated)
+			*minmulti_updated = true;
 		pgcform->relminmxid = minmulti;
 		dirty = true;
 	}
-- 
2.30.2

From 6465bbf843dc8ee549cea5cc6a15c68784098f53 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 22 Nov 2021 10:02:30 -0800
Subject: [PATCH v4 4/5] Decouple advancing relfrozenxid from freezing.

Stop using tuple freezing (and MultiXact freezing) tuple header cutoffs
to determine the final relfrozenxid (and relminmxid) values that we set
for heap relations in pg_class.  Use "optimal" values instead.

Optimal values are the most recent values that are less than or equal to
any remaining XID/MultiXact in a tuple header (not counting frozen
xmin/xmax values).  This is now kept track of by VACUUM.  "Optimal"
values are always >= the tuple header FreezeLimit in an aggressive
VACUUM.  For a non-aggressive VACUUM, they can be less than or greater
than the tuple header FreezeLimit cutoff (though we still often pass
invalid values to indicate that we cannot advance relfrozenxid during
the VACUUM).
---
 src/include/access/heapam.h          |   4 +-
 src/include/access/heapam_xlog.h     |   4 +-
 src/include/commands/vacuum.h        |   1 +
 src/backend/access/heap/heapam.c     | 186 ++++++++++++++++++++-------
 src/backend/access/heap/vacuumlazy.c |  76 +++++++----
 src/backend/commands/cluster.c       |   5 +-
 src/backend/commands/vacuum.c        |  30 ++++-
 7 files changed, 228 insertions(+), 78 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 417dd288e..0eb5c36a2 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -168,7 +168,9 @@ extern bool heap_freeze_tuple(HeapTupleHeader tuple,
 							  TransactionId relfrozenxid, TransactionId relminmxid,
 							  TransactionId cutoff_xid, TransactionId cutoff_multi);
 extern bool heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-									MultiXactId cutoff_multi, Buffer buf);
+									MultiXactId cutoff_multi,
+									TransactionId *NewRelfrozenxid,
+									MultiXactId *NewRelminmxid, Buffer buf);
 extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
 
 extern void simple_heap_insert(Relation relation, HeapTuple tup);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index ab9e873bc..b0ede623e 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -410,7 +410,9 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 									  TransactionId cutoff_xid,
 									  TransactionId cutoff_multi,
 									  xl_heap_freeze_tuple *frz,
-									  bool *totally_frozen);
+									  bool *totally_frozen,
+									  TransactionId *NewRelfrozenxid,
+									  MultiXactId *NewRelminmxid);
 extern void heap_execute_freeze_tuple(HeapTupleHeader tuple,
 									  xl_heap_freeze_tuple *xlrec_tp);
 extern XLogRecPtr log_heap_visible(RelFileNode rnode, Buffer heap_buffer,
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 6eefe8129..114d6da89 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -271,6 +271,7 @@ extern bool vacuum_set_xid_limits(Relation rel,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
+								  MultiXactId *oldestMxact,
 								  TransactionId *freezeLimit,
 								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0b4a46b31..d296a79ea 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6078,12 +6078,24 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
  * FRM_RETURN_IS_MULTI
  *		The return value is a new MultiXactId to set as new Xmax.
  *		(caller must obtain proper infomask bits using GetMultiXactIdHintBits)
+ *
+ * "NewRelfrozenxid" is an output value; it's used to maintain target new
+ * relfrozenxid for the relation.  It can be ignored unless "flags" contains
+ * either FRM_NOOP or FRM_RETURN_IS_MULTI, because we only handle multiXacts
+ * here.  This follows the general convention: only track XIDs that will still
+ * be in the table after the ongoing VACUUM finishes.  Note that it's up to
+ * caller to maintain this when the Xid return value is itself an Xid.
+ *
+ * Note that we cannot depend on xmin to maintain NewRelfrozenxid.  We need to
+ * push maintenance of NewRelfrozenxid down this far, since in general xmin
+ * might have been frozen by an earlier VACUUM operation, in which case our
+ * caller will not have factored-in xmin when maintaining NewRelfrozenxid.
  */
 static TransactionId
 FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 				  TransactionId relfrozenxid, TransactionId relminmxid,
 				  TransactionId cutoff_xid, MultiXactId cutoff_multi,
-				  uint16 *flags)
+				  uint16 *flags, TransactionId *NewRelfrozenxid)
 {
 	TransactionId xid = InvalidTransactionId;
 	int			i;
@@ -6095,6 +6107,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	bool		has_lockers;
 	TransactionId update_xid;
 	bool		update_committed;
+	TransactionId tempNewRelfrozenxid;
 
 	*flags = 0;
 
@@ -6189,13 +6202,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 
 	/* is there anything older than the cutoff? */
 	need_replace = false;
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	for (i = 0; i < nmembers; i++)
 	{
 		if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-		{
 			need_replace = true;
-			break;
-		}
+		if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+			tempNewRelfrozenxid = members[i].xid;
 	}
 
 	/*
@@ -6204,6 +6217,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 */
 	if (!need_replace)
 	{
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 		*flags |= FRM_NOOP;
 		pfree(members);
 		return InvalidTransactionId;
@@ -6213,6 +6227,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 * If the multi needs to be updated, figure out which members do we need
 	 * to keep.
 	 */
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	nnewmembers = 0;
 	newmembers = palloc(sizeof(MultiXactMember) * nmembers);
 	has_lockers = false;
@@ -6294,7 +6309,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			 * list.)
 			 */
 			if (TransactionIdIsValid(update_xid))
+			{
 				newmembers[nnewmembers++] = members[i];
+				if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+					tempNewRelfrozenxid = members[i].xid;
+			}
 		}
 		else
 		{
@@ -6304,6 +6323,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			{
 				/* running locker cannot possibly be older than the cutoff */
 				Assert(!TransactionIdPrecedes(members[i].xid, cutoff_xid));
+				Assert(!TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid));
 				newmembers[nnewmembers++] = members[i];
 				has_lockers = true;
 			}
@@ -6332,6 +6352,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		if (update_committed)
 			*flags |= FRM_MARK_COMMITTED;
 		xid = update_xid;
+		/* Caller manages NewRelfrozenxid directly when we return an XID */
 	}
 	else
 	{
@@ -6341,6 +6362,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		 */
 		xid = MultiXactIdCreateFromMembers(nnewmembers, newmembers);
 		*flags |= FRM_RETURN_IS_MULTI;
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 	}
 
 	pfree(newmembers);
@@ -6359,6 +6381,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
  * will be totally frozen after these operations are performed and false if
  * more freezing will eventually be required.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will actually go on to freeze as indicated by our *frz output, so
+ * any (xmin, xmax, xvac) XIDs that we indicate need to be frozen won't need
+ * to be counted here.  Values are valid lower bounds at the point that the
+ * ongoing VACUUM finishes.
+ *
  * Caller is responsible for setting the offset field, if appropriate.
  *
  * It is assumed that the caller has checked the tuple with
@@ -6383,7 +6412,9 @@ bool
 heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 						  TransactionId relfrozenxid, TransactionId relminmxid,
 						  TransactionId cutoff_xid, TransactionId cutoff_multi,
-						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p)
+						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p,
+						  TransactionId *NewRelfrozenxid,
+						  MultiXactId *NewRelminmxid)
 {
 	bool		changed = false;
 	bool		xmax_already_frozen = false;
@@ -6427,6 +6458,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->t_infomask |= HEAP_XMIN_FROZEN;
 			changed = true;
 		}
+		else if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+		{
+			/* won't be frozen, but older than current NewRelfrozenxid */
+			*NewRelfrozenxid = xid;
+		}
 	}
 
 	/*
@@ -6444,10 +6480,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 	{
 		TransactionId newxmax;
 		uint16		flags;
+		TransactionId temp = *NewRelfrozenxid;
 
 		newxmax = FreezeMultiXactId(xid, tuple->t_infomask,
 									relfrozenxid, relminmxid,
-									cutoff_xid, cutoff_multi, &flags);
+									cutoff_xid, cutoff_multi, &flags, &temp);
 
 		freeze_xmax = (flags & FRM_INVALIDATE_XMAX);
 
@@ -6465,6 +6502,24 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			if (flags & FRM_MARK_COMMITTED)
 				frz->t_infomask |= HEAP_XMAX_COMMITTED;
 			changed = true;
+
+			if (TransactionIdPrecedes(newxmax, *NewRelfrozenxid))
+			{
+				/* New xmax is an XID older than new NewRelfrozenxid */
+				*NewRelfrozenxid = newxmax;
+			}
+		}
+		else if (flags & FRM_NOOP)
+		{
+			/*
+			 * Changing nothing, so might have to ratchet back NewRelminmxid,
+			 * NewRelfrozenxid, or both together
+			 */
+			if (MultiXactIdIsValid(xid) &&
+				MultiXactIdPrecedes(xid, *NewRelminmxid))
+				*NewRelminmxid = xid;
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 		else if (flags & FRM_RETURN_IS_MULTI)
 		{
@@ -6486,6 +6541,13 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->xmax = newxmax;
 
 			changed = true;
+
+			/*
+			 * New multixact might have remaining XID older than
+			 * NewRelfrozenxid
+			 */
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 	}
 	else if (TransactionIdIsNormal(xid))
@@ -6513,7 +6575,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			freeze_xmax = true;
 		}
 		else
+		{
 			freeze_xmax = false;
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			{
+				/* won't be frozen, but older than current NewRelfrozenxid */
+				*NewRelfrozenxid = xid;
+			}
+		}
 	}
 	else if ((tuple->t_infomask & HEAP_XMAX_INVALID) ||
 			 !TransactionIdIsValid(HeapTupleHeaderGetRawXmax(tuple)))
@@ -6560,6 +6629,9 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 		 * was removed in PostgreSQL 9.0.  Note that if we were to respect
 		 * cutoff_xid here, we'd need to make surely to clear totally_frozen
 		 * when we skipped freezing on that basis.
+		 *
+		 * Since we always freeze here, NewRelfrozenxid doesn't need to be
+		 * maintained.
 		 */
 		if (TransactionIdIsNormal(xid))
 		{
@@ -6637,11 +6709,14 @@ heap_freeze_tuple(HeapTupleHeader tuple,
 	xl_heap_freeze_tuple frz;
 	bool		do_freeze;
 	bool		tuple_totally_frozen;
+	TransactionId NewRelfrozenxid = FirstNormalTransactionId;
+	MultiXactId NewRelminmxid = FirstMultiXactId;
 
 	do_freeze = heap_prepare_freeze_tuple(tuple,
 										  relfrozenxid, relminmxid,
 										  cutoff_xid, cutoff_multi,
-										  &frz, &tuple_totally_frozen);
+										  &frz, &tuple_totally_frozen,
+										  &NewRelfrozenxid, &NewRelminmxid);
 
 	/*
 	 * Note that because this is not a WAL-logged operation, we don't need to
@@ -7071,6 +7146,15 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  * Check to see whether any of the XID fields of a tuple (xmin, xmax, xvac)
  * are older than the specified cutoff XID or MultiXactId.  If so, return true.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will never freeze any of the XIDs from the tuple, even when we say
+ * that they should.  If caller opts to go with our recommendation to freeze,
+ * then it must account for the fact that it shouldn't trust how we've set
+ * NewRelfrozenxid/NewRelminmxid.  (In practice aggressive VACUUMs always take
+ * our recommendation because they must, and non-aggressive VACUUMs always opt
+ * to not freeze, preferring to ratchet back NewRelfrozenxid instead).
+ *
  * It doesn't matter whether the tuple is alive or dead, we are checking
  * to see if a tuple needs to be removed or frozen to avoid wraparound.
  *
@@ -7079,74 +7163,86 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  */
 bool
 heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-						MultiXactId cutoff_multi, Buffer buf)
+						MultiXactId cutoff_multi,
+						TransactionId *NewRelfrozenxid,
+						MultiXactId *NewRelminmxid, Buffer buf)
 {
 	TransactionId xid;
+	bool		needs_freeze = false;
 
 	xid = HeapTupleHeaderGetXmin(tuple);
-	if (TransactionIdIsNormal(xid) &&
-		TransactionIdPrecedes(xid, cutoff_xid))
-		return true;
+	if (TransactionIdIsNormal(xid))
+	{
+		if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			*NewRelfrozenxid = xid;
+		if (TransactionIdPrecedes(xid, cutoff_xid))
+			needs_freeze = true;
+	}
 
 	/*
 	 * The considerations for multixacts are complicated; look at
 	 * heap_prepare_freeze_tuple for justifications.  This routine had better
 	 * be in sync with that one!
+	 *
+	 * (Actually, we maintain NewRelminmxid differently here, because we
+	 * assume that XIDs that should be frozen according to cutoff_xid won't
+	 * be, whereas heap_prepare_freeze_tuple makes the opposite assumption.)
 	 */
 	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
 	{
 		MultiXactId multi;
+		MultiXactMember *members;
+		int			nmembers;
 
 		multi = HeapTupleHeaderGetRawXmax(tuple);
-		if (!MultiXactIdIsValid(multi))
-		{
-			/* no xmax set, ignore */
-			;
-		}
-		else if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
+		if (MultiXactIdIsValid(multi) &&
+			MultiXactIdPrecedes(multi, *NewRelminmxid))
+			*NewRelminmxid = multi;
+
+		if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
 			return true;
 		else if (MultiXactIdPrecedes(multi, cutoff_multi))
-			return true;
-		else
+			needs_freeze = true;
+
+		/* need to check whether any member of the mxact is too old */
+		nmembers = GetMultiXactIdMembers(multi, &members, false,
+										 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
+
+		for (int i = 0; i < nmembers; i++)
 		{
-			MultiXactMember *members;
-			int			nmembers;
-			int			i;
-
-			/* need to check whether any member of the mxact is too old */
-
-			nmembers = GetMultiXactIdMembers(multi, &members, false,
-											 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
-
-			for (i = 0; i < nmembers; i++)
-			{
-				if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-				{
-					pfree(members);
-					return true;
-				}
-			}
-			if (nmembers > 0)
-				pfree(members);
+			if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
+				needs_freeze = true;
+			if (TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
 		}
+		if (nmembers > 0)
+			pfree(members);
 	}
 	else
 	{
 		xid = HeapTupleHeaderGetRawXmax(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
 	if (tuple->t_infomask & HEAP_MOVED)
 	{
 		xid = HeapTupleHeaderGetXvac(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
-	return false;
+	return needs_freeze;
 }
 
 /*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index da5b3f79a..c6facc9eb 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -331,8 +331,10 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
-	/* Are FreezeLimit/MultiXactCutoff still valid? */
-	bool		freeze_cutoffs_valid;
+
+	/* Track new pg_class.relfrozenxid/pg_class.relminmxid values */
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -501,6 +503,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
 
@@ -537,8 +540,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 									   params->freeze_table_age,
 									   params->multixact_freeze_min_age,
 									   params->multixact_freeze_table_age,
-									   &OldestXmin, &FreezeLimit,
-									   &MultiXactCutoff);
+									   &OldestXmin, &OldestMxact,
+									   &FreezeLimit, &MultiXactCutoff);
 
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
@@ -602,8 +605,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
-	/* Track if cutoffs became invalid (possible in !aggressive case only) */
-	vacrel->freeze_cutoffs_valid = true;
+
+	/* Initialize values used to advance relfrozenxid/relminmxid at the end */
+	vacrel->NewRelfrozenxid = OldestXmin;
+	vacrel->NewRelminmxid = OldestMxact;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -692,16 +697,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
 	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
 	 * provided we didn't skip any all-visible (not all-frozen) pages using
-	 * the visibility map, and assuming that we didn't fail to get a cleanup
-	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
-	 * MultiXactCutoff) established for VACUUM operation.
+	 * the visibility map.  A non-aggressive VACUUM might only be able to
+	 * advance relfrozenxid to an XID from before FreezeLimit (or a relminmxid
+	 * from before MultiXactCutoff) when it wasn't possible to freeze some
+	 * tuples due to our inability to acquire a cleanup lock, but the effect
+	 * is usually insignificant -- NewRelfrozenxid value still has a decent
+	 * chance of being much more recent that the existing relfrozenxid.
 	 *
 	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
 	 * the rel_pages used by lazy_scan_heap, which won't match when we
 	 * happened to truncate the relation afterwards.
 	 */
-	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
-		!vacrel->freeze_cutoffs_valid)
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
@@ -718,7 +725,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff,
+							vacrel->NewRelfrozenxid, vacrel->NewRelminmxid,
 							&frozenxid_updated, &minmulti_updated, false);
 	}
 
@@ -820,14 +827,14 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 OldestXmin, diff);
 			if (frozenxid_updated)
 			{
-				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				diff = (int32) (vacrel->NewRelfrozenxid - vacrel->relfrozenxid);
 				appendStringInfo(&buf,
 								 _("relfrozenxid: advanced by %d xact IDs, new value: %u\n"),
-								 diff, FreezeLimit);
+								 diff, vacrel->NewRelfrozenxid);
 			}
 			if (minmulti_updated)
 			{
-				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				diff = (int32) (vacrel->NewRelminmxid - vacrel->relminmxid);
 				appendStringInfo(&buf,
 								 _("relminmxid: advanced by %d multixact IDs, new value: %u\n"),
 								 diff, MultiXactCutoff);
@@ -1798,6 +1805,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	int			nfrozen;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1806,6 +1815,8 @@ lazy_scan_prune(LVRelState *vacrel,
 retry:
 
 	/* Initialize (or reset) page-level counters */
+	NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	NewRelminmxid = vacrel->NewRelminmxid;
 	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
@@ -2015,7 +2026,9 @@ retry:
 									  vacrel->FreezeLimit,
 									  vacrel->MultiXactCutoff,
 									  &frozen[nfrozen],
-									  &tuple_totally_frozen))
+									  &tuple_totally_frozen,
+									  &NewRelfrozenxid,
+									  &NewRelminmxid))
 		{
 			/* Will execute freeze below */
 			frozen[nfrozen++].offset = offnum;
@@ -2029,13 +2042,16 @@ retry:
 			prunestate->all_frozen = false;
 	}
 
+	vacrel->offnum = InvalidOffsetNumber;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
 	 * that remains and needs to be considered for freezing now (LP_UNUSED and
 	 * LP_REDIRECT items also remain, but are of no further interest to us).
 	 */
-	vacrel->offnum = InvalidOffsetNumber;
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
 
 	/*
 	 * Consider the need to freeze any items with tuple storage from the page
@@ -2179,9 +2195,9 @@ retry:
  * We'll always return true for a non-aggressive VACUUM, even when we know
  * that this will cause them to miss out on freezing tuples from before
  * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
- * lock.  This does mean that they definitely won't be able to advance
- * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
- * relminmxid).  Caller waits for full cleanup lock when we return false.
+ * lock.  This does mean that they will have NewRelfrozenxid ratcheting back
+ * to a known-safe value (same applies to NewRelminmxid).  Caller waits for
+ * full cleanup lock when we return false.
  *
  * See lazy_scan_prune for an explanation of hastup return flag.  The
  * hasfreespace flag instructs caller on whether or not it should do generic
@@ -2205,6 +2221,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	MultiXactId NewRelminmxid = vacrel->NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -2250,7 +2268,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
 		if (heap_tuple_needs_freeze(tupleheader,
 									vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
+									vacrel->MultiXactCutoff,
+									&NewRelfrozenxid, &NewRelminmxid, buf))
 		{
 			if (vacrel->aggressive)
 			{
@@ -2260,10 +2279,11 @@ lazy_scan_noprune(LVRelState *vacrel,
 			}
 
 			/*
-			 * Current non-aggressive VACUUM operation definitely won't be
-			 * able to advance relfrozenxid or relminmxid
+			 * A non-aggressive VACUUM doesn't have to wait on a cleanup lock
+			 * to ensure that it advances relfrozenxid to a sufficiently
+			 * recent XID that happens to be present on this page.  It can
+			 * just accept an older New/final relfrozenxid instead.
 			 */
-			vacrel->freeze_cutoffs_valid = false;
 		}
 
 		num_tuples++;
@@ -2313,6 +2333,14 @@ lazy_scan_noprune(LVRelState *vacrel,
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * We have committed to not freezing the tuples on this page (always
+	 * happens with a non-aggressive VACUUM), so make sure that the target
+	 * relfrozenxid/relminmxid values reflect the XIDs/MXIDs we encountered
+	 */
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
+
 	/*
 	 * Now save details of the LP_DEAD items from the page in vacrel (though
 	 * only when VACUUM uses two-pass strategy).
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 66b87347d..6bd6688ae 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -767,6 +767,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	TupleDesc	oldTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TupleDesc	newTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TransactionId OldestXmin;
+	MultiXactId oldestMxact;
 	TransactionId FreezeXid;
 	MultiXactId MultiXactCutoff;
 	bool		use_sort;
@@ -856,8 +857,8 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * Since we're going to rewrite the whole table anyway, there's no reason
 	 * not to be aggressive about this.
 	 */
-	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
+	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0, &OldestXmin, &oldestMxact,
+						  &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 6db7b8156..cce290c78 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -943,10 +943,28 @@ get_all_vacuum_rels(int options)
  * The output parameters are:
  * - oldestXmin is the Xid below which tuples deleted by any xact (that
  *   committed) should be considered DEAD, not just RECENTLY_DEAD.
+ * - oldestMxact is the Mxid below which MultiXacts are definitely not
+ *   seen as visible by any running transaction.
  * - freezeLimit is the Xid below which all Xids are replaced by
  *	 FrozenTransactionId during vacuum.
  * - multiXactCutoff is the value below which all MultiXactIds are removed
  *   from Xmax.
+ *
+ * oldestXmin and oldestMxact can be thought of as the most recent values that
+ * can ever be passed to vac_update_relstats() as frozenxid and minmulti
+ * arguments.  These exact values will be used when no newer XIDs or
+ * MultiXacts remain in the heap relation (e.g., with an empty table).  It's
+ * typical for vacuumlazy.c caller to notice that older XIDs/Multixacts remain
+ * in the table, which will force it to use older value.  These older final
+ * values may not be any newer than the preexisting frozenxid/minmulti values
+ * from pg_class in extreme cases.  The final values are frequently fairly
+ * close to the optimal values that we give to vacuumlazy.c, though.
+ *
+ * An aggressive VACUUM always provides vac_update_relstats() arguments that
+ * are >= freezeLimit and >= multiXactCutoff.  A non-aggressive VACUUM may
+ * provide arguments that are either newer or older than freezeLimit and
+ * multiXactCutoff, or non-valid values (indicating that pg_class level
+ * cutoffs cannot be advanced at all).
  */
 bool
 vacuum_set_xid_limits(Relation rel,
@@ -955,6 +973,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_min_age,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
+					  MultiXactId *oldestMxact,
 					  TransactionId *freezeLimit,
 					  MultiXactId *multiXactCutoff)
 {
@@ -963,7 +982,6 @@ vacuum_set_xid_limits(Relation rel,
 	int			effective_multixact_freeze_max_age;
 	TransactionId limit;
 	TransactionId safeLimit;
-	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
 	int			freezetable;
@@ -1059,9 +1077,11 @@ vacuum_set_xid_limits(Relation rel,
 						 effective_multixact_freeze_max_age / 2);
 	Assert(mxid_freezemin >= 0);
 
+	/* Remember for caller */
+	*oldestMxact = GetOldestMultiXactId();
+
 	/* compute the cutoff multi, being careful to generate a valid value */
-	oldestMxact = GetOldestMultiXactId();
-	mxactLimit = oldestMxact - mxid_freezemin;
+	mxactLimit = *oldestMxact - mxid_freezemin;
 	if (mxactLimit < FirstMultiXactId)
 		mxactLimit = FirstMultiXactId;
 
@@ -1076,8 +1096,8 @@ vacuum_set_xid_limits(Relation rel,
 				(errmsg("oldest multixact is far in the past"),
 				 errhint("Close open transactions with multixacts soon to avoid wraparound problems.")));
 		/* Use the safe limit, unless an older mxact is still running */
-		if (MultiXactIdPrecedes(oldestMxact, safeMxactLimit))
-			mxactLimit = oldestMxact;
+		if (MultiXactIdPrecedes(*oldestMxact, safeMxactLimit))
+			mxactLimit = *oldestMxact;
 		else
 			mxactLimit = safeMxactLimit;
 	}
-- 
2.30.2

From 919bd29902da15a5d43166dabf379cdc60d7dacb Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 13 Dec 2021 15:00:49 -0800
Subject: [PATCH v4 5/5] Prototype of opportunistic freezing.

Freeze whenever pruning modified the page, or whenever we see that we're
going to mark the page all-visible without also marking it all-frozen.

There has been plenty of discussion of opportunistic freezing in the
past.  It is generally considered important as a way of minimizing
repeated dirtying of heap pages (or the total volume of FPIs in the WAL
stream) over time.  While that goal is certainly very important, this
patch has another priority: making VACUUM advance relfrozenxid sooner
and more frequently.

The overall effect is that tables like pgbench's history table can be
vacuumed very frequently, and have most individual vacuum operations
generate 0 FPIs in WAL -- they will never need an aggressive VACUUM.
The old SKIP_PAGES_THRESHOLD heuristic is designed to make it more
likely that we'll be able to advance relfrozenxid, which works well when
combined with additions from earlier patches in the patch series, and
opportunistic freezing.

GUCs like vacuum_freeze_min_age never made much sense after the freeze
map work in PostgreSQL 9.6.  The default is 50 million transactions,
which current tends to result in our being unable to freeze tuples
before the page is marked all-visible (but not all-frozen).  This
creates a huge performance cliff later on, during the first aggressive
VACUUM.  And so an important goal of opportunistic freezing is to not
allow the system to get into too much "debt" from very old unfrozen
tuples.  That might actually be more important than minimizing the
absolute cost of freezing.

There is probably a small regression caused by opportunistic freezing
with workloads like pgbench, since we're freezing many more tuples than
we need to now -- while we do have fewer FPIs (even earlier on), that
may not be enough to make up for the increase in WAL records.  This
problem can be addressed in a later revision, when the general picture
for this patch (especially how it affects our ability to advance
relfrozenxid early) becomes clearer.
---
 src/include/access/heapam.h          |  1 +
 src/backend/access/heap/pruneheap.c  |  8 ++-
 src/backend/access/heap/vacuumlazy.c | 78 +++++++++++++++++++++++++---
 3 files changed, 79 insertions(+), 8 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 0eb5c36a2..5e1f24e5c 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -188,6 +188,7 @@ extern int	heap_page_prune(Relation relation, Buffer buffer,
 							struct GlobalVisState *vistest,
 							TransactionId old_snap_xmin,
 							TimestampTz old_snap_ts_ts,
+							bool *modified,
 							int	*nnewlpdead,
 							OffsetNumber *off_loc);
 extern void heap_page_prune_execute(Buffer buffer,
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 522a00af6..e95dea38d 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -182,11 +182,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
 		 */
 		if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
 		{
+			bool	modified;
 			int		ndeleted,
 					nnewlpdead;
 
 			ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
-									   limited_ts, &nnewlpdead, NULL);
+									   limited_ts, &modified, &nnewlpdead, NULL);
 
 			/*
 			 * Report the number of tuples reclaimed to pgstats.  This is
@@ -244,6 +245,7 @@ heap_page_prune(Relation relation, Buffer buffer,
 				GlobalVisState *vistest,
 				TransactionId old_snap_xmin,
 				TimestampTz old_snap_ts,
+				bool *modified,
 				int	*nnewlpdead,
 				OffsetNumber *off_loc)
 {
@@ -375,6 +377,8 @@ heap_page_prune(Relation relation, Buffer buffer,
 
 			PageSetLSN(BufferGetPage(buffer), recptr);
 		}
+
+		*modified = true;
 	}
 	else
 	{
@@ -387,12 +391,14 @@ heap_page_prune(Relation relation, Buffer buffer,
 		 * point in repeating the prune/defrag process until something else
 		 * happens to the page.
 		 */
+		*modified = false;
 		if (((PageHeader) page)->pd_prune_xid != prstate.new_prune_xid ||
 			PageIsFull(page))
 		{
 			((PageHeader) page)->pd_prune_xid = prstate.new_prune_xid;
 			PageClearFull(page);
 			MarkBufferDirtyHint(buffer, true);
+			*modified = true;
 		}
 	}
 
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index c6facc9eb..a710c6cf8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -328,6 +328,7 @@ typedef struct LVRelState
 
 	/* VACUUM operation's cutoff for pruning */
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
@@ -529,11 +530,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/*
 	 * Get cutoffs that determine which tuples we need to freeze during the
-	 * VACUUM operation.
+	 * VACUUM operation.  This includes information that is used during
+	 * opportunistic freezing, where the most aggressive possible cutoffs
+	 * (OldestXmin and OldestMxact) are used for some heap pages, based on
+	 * considerations about cost.
 	 *
 	 * Also determines if this is to be an aggressive VACUUM.  This will
 	 * eventually be required for any table where (for whatever reason) no
 	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
+	 * This used to be much more common, but we now work hard to advance
+	 * relfrozenxid in non-aggressive VACUUMs.
 	 */
 	aggressive = vacuum_set_xid_limits(rel,
 									   params->freeze_min_age,
@@ -603,6 +609,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/* Set cutoffs for entire VACUUM */
 	vacrel->OldestXmin = OldestXmin;
+	vacrel->OldestMxact = OldestMxact;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
 
@@ -1807,6 +1814,10 @@ lazy_scan_prune(LVRelState *vacrel,
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 	TransactionId NewRelfrozenxid;
 	MultiXactId NewRelminmxid;
+	bool		modified;
+	TransactionId FreezeLimit = vacrel->FreezeLimit;
+	MultiXactId MultiXactCutoff = vacrel->MultiXactCutoff;
+	bool		earlyfreezing = false;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1833,8 +1844,19 @@ retry:
 	 * that were deleted from indexes.
 	 */
 	tuples_deleted = heap_page_prune(rel, buf, vistest,
-									 InvalidTransactionId, 0, &nnewlpdead,
-									 &vacrel->offnum);
+									 InvalidTransactionId, 0, &modified,
+									 &nnewlpdead, &vacrel->offnum);
+
+	/*
+	 * If page was modified during pruning, then perform early freezing
+	 * opportunistically
+	 */
+	if (!earlyfreezing && modified)
+	{
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+	}
 
 	/*
 	 * Now scan the page to collect LP_DEAD items and check for tuples
@@ -1889,7 +1911,7 @@ retry:
 		if (ItemIdIsDead(itemid))
 		{
 			deadoffsets[lpdead_items++] = offnum;
-			prunestate->all_visible = false;
+			/* Don't set all_visible to false just yet */
 			prunestate->has_lpdead_items = true;
 			continue;
 		}
@@ -2023,8 +2045,8 @@ retry:
 		if (heap_prepare_freeze_tuple(tuple.t_data,
 									  vacrel->relfrozenxid,
 									  vacrel->relminmxid,
-									  vacrel->FreezeLimit,
-									  vacrel->MultiXactCutoff,
+									  FreezeLimit,
+									  MultiXactCutoff,
 									  &frozen[nfrozen],
 									  &tuple_totally_frozen,
 									  &NewRelfrozenxid,
@@ -2044,6 +2066,48 @@ retry:
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * If page is going to become all_visible (excluding any LP_DEAD items),
+	 * but won't also become all_frozen (either in ongoing first heap pass, or
+	 * in second heap pass after LP_DEAD items get set LP_UNUSED) then repeat
+	 * our pass over the heap, using more aggressive (opportunistic) freeze
+	 * limits.  This policy isn't guaranteed to be cheaper in the long run,
+	 * but it often is.  And it makes it far more likely that non-aggressive
+	 * VACUUMs will end up advancing relfrozenxid to a reasonably recent XID;
+	 * an XID that we opt to freeze won't hold back NewRelfrozenxid.
+	 *
+	 * We deliberately track all_visible in a way that excludes LP_DEAD items
+	 * here.  Our assumption is that any page that is "all_visible for tuples
+	 * with storage" will be safe to mark all_visible in the visibility map
+	 * during VACUUM's second heap pass, right after LP_DEAD items are set
+	 * LP_UNUSED.  Either way (with or without LP_DEAD items), our goal is to
+	 * ensure that a page that _would have_ been marked all_visible in the
+	 * visibility map gets marked all_frozen instead.
+	 */
+	if (!earlyfreezing && prunestate->all_visible && !prunestate->all_frozen)
+	{
+		/*
+		 * XXX Need to worry about leaking MultiXacts in FreezeMultiXactId()
+		 * now (via heap_prepare_freeze_tuple calls)?  That was already
+		 * possible, but presumably this makes it much more likely.
+		 *
+		 * On the other hand, that's only possible when we need to replace an
+		 * existing MultiXact with a new one.  Even then, we won't have
+		 * preallocated a new MultiXact (which we now risk leaking) if there
+		 * was only one remaining XID, and the XID is for an updater (we'll
+		 * only prepare to replace xmax with the XID directly).  So maybe it's
+		 * still a narrow enough problem to be ignored.
+		 */
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+		goto retry;
+	}
+
+	/* Time to define all_visible in a way that accounts for LP_DEAD items */
+	if (lpdead_items > 0)
+		prunestate->all_visible = false;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
@@ -2089,7 +2153,7 @@ retry:
 		{
 			XLogRecPtr	recptr;
 
-			recptr = log_heap_freeze(vacrel->rel, buf, vacrel->FreezeLimit,
+			recptr = log_heap_freeze(vacrel->rel, buf, FreezeLimit,
 									 frozen, nfrozen);
 			PageSetLSN(page, recptr);
 		}
-- 
2.30.2

From 6b4b69741461e60e66a5fcf6673ffb3e87aed1ae Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v4 1/5] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

We now also collect LP_DEAD items in the dead_items array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

We no longer report on "pin skipped pages" in log output.  A later patch
will add back an improved version of the same instrumentation.  We don't
want to show any information about any failures to acquire cleanup locks
unless we actually failed to do useful work as a consequence.  A page
that we could not acquire a cleanup lock on is now treated as equivalent
to any other scanned page in most cases.
---
 src/backend/access/heap/vacuumlazy.c          | 814 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 516 insertions(+), 307 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index db6becfed..c6d3a483f 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -152,7 +152,7 @@ typedef enum
 /*
  * LVDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
  * Each TID points to an LP_DEAD line pointer from a heap page that has been
- * processed by lazy_scan_prune.
+ * processed by lazy_scan_prune (or by lazy_scan_noprune, perhaps).
  *
  * Also needed by lazy_vacuum_heap_rel, which marks the same LP_DEAD line
  * pointers as LP_UNUSED during second heap pass.
@@ -305,6 +305,8 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -329,6 +331,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -343,10 +347,8 @@ typedef struct LVRelState
 	 */
 	LVDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -359,6 +361,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -398,19 +401,22 @@ static int	elevel = -1;
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, bool skipwithvm, int nworkers);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup, bool *hasfreespace);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void lazy_cleanup_all_indexes(LVRelState *vacrel);
 static void parallel_vacuum_process_all_indexes(LVRelState *vacrel, bool vacuum);
@@ -480,16 +486,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive,
+				skipwithvm;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -535,8 +540,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 											   xidFullScanLimit);
 	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
 											  mxactFullScanLimit);
+	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
+	{
+		/*
+		 * Force aggressive mode, and disable skipping blocks using the
+		 * visibility map (even those set all-frozen)
+		 */
 		aggressive = true;
+		skipwithvm = false;
+	}
 
 	vacrel = (LVRelState *) palloc0(sizeof(LVRelState));
 
@@ -544,6 +557,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->rel = rel;
 	vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
 					 &vacrel->indrels);
+	vacrel->aggressive = aggressive;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 
@@ -588,6 +602,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -624,30 +640,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, skipwithvm, params->nworkers);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		/*
@@ -678,28 +680,44 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
+	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
+	 * provided we didn't skip any all-visible (not all-frozen) pages using
+	 * the visibility map, and assuming that we didn't fail to get a cleanup
+	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
+	 * MultiXactCutoff) established for VACUUM operation.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_heap, which won't match when we
+	 * happened to truncate the relation afterwards.
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -728,7 +746,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -775,10 +792,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -786,7 +802,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->pages_removed;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -903,7 +918,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, bool skipwithvm, int nworkers)
 {
 	LVDeadItems *dead_items;
 	BlockNumber nblocks,
@@ -925,7 +940,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	pg_rusage_init(&ru0);
 
-	if (aggressive)
+	if (vacrel->aggressive)
 		ereport(elevel,
 				(errmsg("aggressively vacuuming \"%s.%s\"",
 						vacrel->relnamespace,
@@ -937,14 +952,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 						vacrel->relname)));
 
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
-	next_unskippable_block = 0;
-	next_failsafe_block = 0;
-	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -968,14 +978,16 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * dangerously old.
 	 */
 	lazy_check_wraparound_failsafe(vacrel);
+	next_failsafe_block = 0;
 
 	/*
 	 * Allocate the space for dead_items.  Note that this handles parallel
 	 * VACUUM initialization as part of allocating shared memory space used
 	 * for dead_items.
 	 */
-	dead_items_alloc(vacrel, params->nworkers);
+	dead_items_alloc(vacrel, nworkers);
 	dead_items = vacrel->dead_items;
+	next_fsm_block_to_vacuum = 0;
 
 	/* Report that we're scanning the heap, advertising total # of blocks */
 	initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
@@ -984,7 +996,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/*
-	 * Except when aggressive is set, we want to skip pages that are
+	 * Set things up for skipping blocks using visibility map.
+	 *
+	 * Except when vacrel->aggressive is set, we want to skip pages that are
 	 * all-visible according to the visibility map, but only when we can skip
 	 * at least SKIP_PAGES_THRESHOLD consecutive pages.  Since we're reading
 	 * sequentially, the OS should be doing readahead for us, so there's no
@@ -993,8 +1007,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * page means that we can't update relfrozenxid, so we only want to do it
 	 * if we can skip a goodly number of pages.
 	 *
-	 * When aggressive is set, we can't skip pages just because they are
-	 * all-visible, but we can still skip pages that are all-frozen, since
+	 * When vacrel->aggressive is set, we can't skip pages just because they
+	 * are all-visible, but we can still skip pages that are all-frozen, since
 	 * such pages do not need freezing and do not affect the value that we can
 	 * safely set for relfrozenxid or relminmxid.
 	 *
@@ -1017,17 +1031,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
-	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+	next_unskippable_block = 0;
+	if (skipwithvm)
 	{
 		while (next_unskippable_block < nblocks)
 		{
@@ -1036,7 +1042,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vmstatus = visibilitymap_get_status(vacrel->rel,
 												next_unskippable_block,
 												&vmbuffer);
-			if (aggressive)
+			if (vacrel->aggressive)
 			{
 				if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
 					break;
@@ -1063,13 +1069,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
@@ -1079,7 +1078,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		{
 			/* Time to advance next_unskippable_block */
 			next_unskippable_block++;
-			if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+			if (skipwithvm)
 			{
 				while (next_unskippable_block < nblocks)
 				{
@@ -1088,7 +1087,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 					vmskipflags = visibilitymap_get_status(vacrel->rel,
 														   next_unskippable_block,
 														   &vmbuffer);
-					if (aggressive)
+					if (vacrel->aggressive)
 					{
 						if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
 							break;
@@ -1117,19 +1116,25 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			 * it's not all-visible.  But in an aggressive vacuum we know only
 			 * that it's not all-frozen, so it might still be all-visible.
 			 */
-			if (aggressive && VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
+			if (vacrel->aggressive &&
+				VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
 				all_visible_according_to_vm = true;
 		}
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current page can be skipped if we've seen a long enough run
+			 * of skippable blocks to justify skipping it -- provided it's not
+			 * the last page in the relation (according to rel_pages/nblocks).
+			 *
+			 * We always scan the table's last page to determine whether it
+			 * has tuples or not, even if it would otherwise be skipped
+			 * (unless we're skipping every single page in the relation). This
+			 * avoids having lazy_truncate_heap() take access-exclusive lock
+			 * on the table to attempt a truncation that just fails
+			 * immediately because there are tuples on the last page.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks && blkno < nblocks - 1)
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -1138,18 +1143,32 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * careful to count it as a skipped all-frozen page in that
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
-				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * know whether it was initially all-frozen, so we have to
+				 * recheck.
 				 */
-				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+				if (vacrel->aggressive ||
+					VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise it must be an all-visible (and possibly even
+			 * all-frozen) page that we decided to process regardless
+			 * (SKIP_PAGES_THRESHOLD must not have been crossed).
+			 */
 			all_visible_according_to_vm = true;
 		}
 
 		vacuum_delay_point();
 
+		/*
+		 * We're not skipping this page using the visibility map, and so it is
+		 * (by definition) a scanned page.  Any tuples from this page are now
+		 * guaranteed to be counted below, after some preparatory checks.
+		 */
+		vacrel->scanned_pages++;
+
 		/*
 		 * Regularly check if wraparound failsafe should trigger.
 		 *
@@ -1204,174 +1223,78 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
-		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
-		 * already have the correct page pinned anyway.  However, it's
-		 * possible that (a) next_unskippable_block is covered by a different
-		 * VM page than the current block or (b) we released our pin and did a
-		 * cycle of index vacuuming.
+		 * already have the correct page pinned anyway.
 		 */
 		visibilitymap_pin(vacrel->rel, blkno, &vmbuffer);
 
+		/* Finished preparatory checks.  Actually scan the page. */
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing using lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
-			bool		hastup;
+			bool		hastup,
+						hasfreespace;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
-			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				continue;
-			}
-
-			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
-			 */
 			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
+				/* Processed as new/empty page (lock and pin released) */
 				continue;
 			}
-			if (!aggressive)
+
+			/* Collect LP_DEAD items in dead_items array, count tuples */
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
+								  &hasfreespace))
 			{
+				Size		freespace;
+
 				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
+				 * Processed page successfully (without cleanup lock) -- just
+				 * need to perform rel truncation and FSM steps, much like the
+				 * lazy_scan_prune case.  Don't bother trying to match its
+				 * visibility map setting steps, though.
 				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
 				if (hastup)
 					vacrel->nonempty_pages = blkno + 1;
+				if (hasfreespace)
+					freespace = PageGetHeapFreeSpace(page);
+				UnlockReleaseBuffer(buf);
+				if (hasfreespace)
+					RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
 			}
+
+			/*
+			 * lazy_scan_noprune could not do all required processing.  Wait
+			 * for a cleanup lock, and call lazy_scan_prune in the usual way.
+			 */
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Processed as new/empty page (lock and pin released) */
 			continue;
 		}
 
 		/*
-		 * Prune and freeze tuples.
+		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
 		 * dead_items array.  This includes LP_DEAD line pointers that we
@@ -1579,7 +1502,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1652,14 +1575,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
 					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
-	appendStringInfo(&buf, ngettext("Skipped %u page due to buffer pins, ",
-									"Skipped %u pages due to buffer pins, ",
-									vacrel->pinskipped_pages),
-					 vacrel->pinskipped_pages);
-	appendStringInfo(&buf, ngettext("%u frozen page.\n",
-									"%u frozen pages.\n",
-									vacrel->frozenskipped_pages),
-					 vacrel->frozenskipped_pages);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1673,6 +1588,138 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pfree(buf.data);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller must hold at least a shared lock.  We might need to escalate the
+ * lock in that case, so the type of lock caller holds needs to be specified
+ * using 'sharelock' argument.
+ *
+ * Returns false in common case where caller should go on to call
+ * lazy_scan_prune (or lazy_scan_noprune).  Otherwise returns true, indicating
+ * that lazy_scan_heap is done processing the page, releasing lock on caller's
+ * behalf.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never discover the space on a promoted standby.
+		 * The harm of repeated checking ought to normally not be too bad. The
+		 * space usually should be used at some point, otherwise there
+		 * wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1717,6 +1764,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
 	maxoff = PageGetMaxOffsetNumber(page);
 
 retry:
@@ -1779,10 +1828,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -2069,6 +2117,225 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() variant without pruning
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED in lazy_vacuum_heap_page.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).
+ *
+ * Returns true to indicate that all required processing has been performed.
+ * We'll always return true for a non-aggressive VACUUM, even when we know
+ * that this will cause them to miss out on freezing tuples from before
+ * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
+ * lock.  This does mean that they definitely won't be able to advance
+ * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
+ * relminmxid).  Caller waits for full cleanup lock when we return false.
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.  The
+ * hasfreespace flag instructs caller on whether or not it should do generic
+ * FSM processing for page, which is determined based on almost the same
+ * criteria as the lazy_scan_prune case.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup,
+				  bool *hasfreespace)
+{
+	OffsetNumber offnum,
+				maxoff;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
+	*hastup = false;			/* for now */
+	*hasfreespace = false;		/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;		/* page prevents rel truncation */
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (heap_tuple_needs_freeze(tupleheader,
+									vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			/*
+			 * Current non-aggressive VACUUM operation definitely won't be
+			 * able to advance relfrozenxid or relminmxid
+			 */
+			vacrel->freeze_cutoffs_valid = false;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+
+				/*
+				 * There is some useful work for pruning to do, that won't be
+				 * done due to failure to get a cleanup lock.
+				 *
+				 * TODO Add dedicated instrumentation for this case
+				 */
+				break;
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in vacrel (though
+	 * only when VACUUM uses two-pass strategy).
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/*
+		 * Using one-pass strategy.
+		 *
+		 * We are not prepared to handle the corner case where a single pass
+		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
+		 * items.
+		 */
+		if (lpdead_items > 0)
+			*hastup = true;
+		*hasfreespace = true;
+		num_tuples += lpdead_items;
+		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+	}
+	else if (lpdead_items > 0)
+	{
+		LVDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * Caller won't be vacuuming this page later, so tell it to record
+		 * page's freespace in the FSM now
+		 */
+		*hasfreespace = true;
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Remove the collected garbage tuples from the table and its indexes.
  *
@@ -2515,67 +2782,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2663,7 +2869,7 @@ parallel_vacuum_process_all_indexes(LVRelState *vacrel, bool vacuum)
 		 */
 		vacrel->lps->lvshared->reltuples = vacrel->new_rel_tuples;
 		vacrel->lps->lvshared->estimated_count =
-			(vacrel->tupcount_pages < vacrel->rel_pages);
+			(vacrel->scanned_pages < vacrel->rel_pages);
 
 		new_status = PARALLEL_INDVAC_STATUS_NEED_CLEANUP;
 
@@ -2982,7 +3188,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -3133,7 +3339,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From 09dff0737d045713cc536c736b8b05b578bfdb9d Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 11 Dec 2021 17:39:45 -0800
Subject: [PATCH v4 3/5] Simplify vacuum_set_xid_limits signature.

Refactoring, making the return value of vacuum_set_xid_limits()
determine whether or not this will be an aggressive VACUUM.

This will make it easier to set/return an oldestMxact value for
vacuumlazy.c caller in the next commit, which is an important detail
that enables advancing relminmxid opportunistically.
---
 src/include/commands/vacuum.h        |   6 +-
 src/backend/access/heap/vacuumlazy.c |  32 +++----
 src/backend/commands/cluster.c       |   3 +-
 src/backend/commands/vacuum.c        | 134 +++++++++++++--------------
 4 files changed, 79 insertions(+), 96 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index bc625463e..6eefe8129 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -266,15 +266,13 @@ extern void vac_update_relstats(Relation relation,
 								bool *frozenxid_updated,
 								bool *minmulti_updated,
 								bool in_outer_xact);
-extern void vacuum_set_xid_limits(Relation rel,
+extern bool vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
 								  TransactionId *freezeLimit,
-								  TransactionId *xidFullScanLimit,
-								  MultiXactId *multiXactCutoff,
-								  MultiXactId *mxactFullScanLimit);
+								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
 									  MultiXactId relminmxid);
 extern void vac_update_datfrozenxid(void);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 238e07a78..da5b3f79a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -494,8 +494,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
-	TransactionId xidFullScanLimit;
-	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
@@ -526,24 +524,22 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
 								  RelationGetRelid(rel));
 
-	vacuum_set_xid_limits(rel,
-						  params->freeze_min_age,
-						  params->freeze_table_age,
-						  params->multixact_freeze_min_age,
-						  params->multixact_freeze_table_age,
-						  &OldestXmin, &FreezeLimit, &xidFullScanLimit,
-						  &MultiXactCutoff, &mxactFullScanLimit);
-
 	/*
-	 * We request an aggressive scan if the table's frozen Xid is now older
-	 * than or equal to the requested Xid full-table scan limit; or if the
-	 * table's minimum MultiXactId is older than or equal to the requested
-	 * mxid full-table scan limit; or if DISABLE_PAGE_SKIPPING was specified.
+	 * Get cutoffs that determine which tuples we need to freeze during the
+	 * VACUUM operation.
+	 *
+	 * Also determines if this is to be an aggressive VACUUM.  This will
+	 * eventually be required for any table where (for whatever reason) no
+	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
 	 */
-	aggressive = TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
-											   xidFullScanLimit);
-	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
-											  mxactFullScanLimit);
+	aggressive = vacuum_set_xid_limits(rel,
+									   params->freeze_min_age,
+									   params->freeze_table_age,
+									   params->multixact_freeze_min_age,
+									   params->multixact_freeze_table_age,
+									   &OldestXmin, &FreezeLimit,
+									   &MultiXactCutoff);
+
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
 	{
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 9d22f648a..66b87347d 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -857,8 +857,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * not to be aggressive about this.
 	 */
 	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, NULL, &MultiXactCutoff,
-						  NULL);
+						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 8bd4bd12c..6db7b8156 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -935,25 +935,20 @@ get_all_vacuum_rels(int options)
  *
  * Input parameters are the target relation, applicable freeze age settings.
  *
+ * Return value indicates whether caller should do an aggressive VACUUM or
+ * not.  This is a VACUUM that cannot skip any pages using the visibility map
+ * (except all-frozen pages), which is guaranteed to be able to advance
+ * relfrozenxid and relminmxid.
+ *
  * The output parameters are:
- * - oldestXmin is the cutoff value used to distinguish whether tuples are
- *	 DEAD or RECENTLY_DEAD (see HeapTupleSatisfiesVacuum).
+ * - oldestXmin is the Xid below which tuples deleted by any xact (that
+ *   committed) should be considered DEAD, not just RECENTLY_DEAD.
  * - freezeLimit is the Xid below which all Xids are replaced by
  *	 FrozenTransactionId during vacuum.
- * - xidFullScanLimit (computed from freeze_table_age parameter)
- *	 represents a minimum Xid value; a table whose relfrozenxid is older than
- *	 this will have a full-table vacuum applied to it, to freeze tuples across
- *	 the whole table.  Vacuuming a table younger than this value can use a
- *	 partial scan.
- * - multiXactCutoff is the value below which all MultiXactIds are removed from
- *	 Xmax.
- * - mxactFullScanLimit is a value against which a table's relminmxid value is
- *	 compared to produce a full-table vacuum, as with xidFullScanLimit.
- *
- * xidFullScanLimit and mxactFullScanLimit can be passed as NULL if caller is
- * not interested.
+ * - multiXactCutoff is the value below which all MultiXactIds are removed
+ *   from Xmax.
  */
-void
+bool
 vacuum_set_xid_limits(Relation rel,
 					  int freeze_min_age,
 					  int freeze_table_age,
@@ -961,9 +956,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
 					  TransactionId *freezeLimit,
-					  TransactionId *xidFullScanLimit,
-					  MultiXactId *multiXactCutoff,
-					  MultiXactId *mxactFullScanLimit)
+					  MultiXactId *multiXactCutoff)
 {
 	int			freezemin;
 	int			mxid_freezemin;
@@ -973,6 +966,7 @@ vacuum_set_xid_limits(Relation rel,
 	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
+	int			freezetable;
 
 	/*
 	 * We can always ignore processes running lazy vacuum.  This is because we
@@ -1090,64 +1084,60 @@ vacuum_set_xid_limits(Relation rel,
 
 	*multiXactCutoff = mxactLimit;
 
-	if (xidFullScanLimit != NULL)
-	{
-		int			freezetable;
+	/*
+	 * Done setting output parameters; just need to figure out if caller needs
+	 * to do an aggressive VACUUM or not.
+	 *
+	 * Determine the table freeze age to use: as specified by the caller, or
+	 * vacuum_freeze_table_age, but in any case not more than
+	 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
+	 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
+	 * before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_freeze_table_age;
+	freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		Assert(mxactFullScanLimit != NULL);
+	/*
+	 * Compute XID limit causing an aggressive vacuum, being careful not to
+	 * generate a "permanent" XID
+	 */
+	limit = ReadNextTransactionId() - freezetable;
+	if (!TransactionIdIsNormal(limit))
+		limit = FirstNormalTransactionId;
+	if (TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
+									  limit))
+		return true;
 
-		/*
-		 * Determine the table freeze age to use: as specified by the caller,
-		 * or vacuum_freeze_table_age, but in any case not more than
-		 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
-		 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
-		 * before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_freeze_table_age;
-		freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
+	/*
+	 * Similar to the above, determine the table freeze age to use for
+	 * multixacts: as specified by the caller, or
+	 * vacuum_multixact_freeze_table_age, but in any case not more than
+	 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have e.g.
+	 * nightly VACUUM schedule, the nightly VACUUM gets a chance to freeze
+	 * multixacts before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = multixact_freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_multixact_freeze_table_age;
+	freezetable = Min(freezetable,
+					  effective_multixact_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		/*
-		 * Compute XID limit causing a full-table vacuum, being careful not to
-		 * generate a "permanent" XID.
-		 */
-		limit = ReadNextTransactionId() - freezetable;
-		if (!TransactionIdIsNormal(limit))
-			limit = FirstNormalTransactionId;
+	/*
+	 * Compute MultiXact limit causing an aggressive vacuum, being careful to
+	 * generate a valid MultiXact value
+	 */
+	mxactLimit = ReadNextMultiXactId() - freezetable;
+	if (mxactLimit < FirstMultiXactId)
+		mxactLimit = FirstMultiXactId;
+	if (MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
+									mxactLimit))
+		return true;
 
-		*xidFullScanLimit = limit;
-
-		/*
-		 * Similar to the above, determine the table freeze age to use for
-		 * multixacts: as specified by the caller, or
-		 * vacuum_multixact_freeze_table_age, but in any case not more than
-		 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have
-		 * e.g. nightly VACUUM schedule, the nightly VACUUM gets a chance to
-		 * freeze multixacts before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = multixact_freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_multixact_freeze_table_age;
-		freezetable = Min(freezetable,
-						  effective_multixact_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
-
-		/*
-		 * Compute MultiXact limit causing a full-table vacuum, being careful
-		 * to generate a valid MultiXact value.
-		 */
-		mxactLimit = ReadNextMultiXactId() - freezetable;
-		if (mxactLimit < FirstMultiXactId)
-			mxactLimit = FirstMultiXactId;
-
-		*mxactFullScanLimit = mxactLimit;
-	}
-	else
-	{
-		Assert(mxactFullScanLimit == NULL);
-	}
+	return false;
 }
 
 /*
-- 
2.30.2

From ee88cec87714420f195bc05d3a1e154c1aa759d6 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 21 Nov 2021 14:47:11 -0800
Subject: [PATCH v5 2/5] Improve log_autovacuum_min_duration output.

Add instrumentation of "missed dead tuples", and the number of pages
that had at least one such tuple.  These are fully DEAD (not just
RECENTLY_DEAD) tuples with storage that could not be pruned due to an
inability to acquire a cleanup lock.  This is a replacement for the
"skipped due to pin" instrumentation removed by the previous commit.
Note that the new instrumentation doesn't say anything about pages that
we failed to acquire a cleanup lock on when we see that there were no
missed dead tuples on the page.

Also report on visibility map pages skipped by VACUUM, without regard
for whether the pages were all-frozen or just all-visible.

Also report when and how relfrozenxid is advanced by VACUUM, including
non-aggressive VACUUM.  Apart from being useful on its own, this might
enable future work that teaches non-aggressive VACUUM to be more
concerned about advancing relfrozenxid sooner rather than later.

Also report number of tuples frozen.  This will become more important
when the later patch to perform opportunistic tuple freezing is
committed.

Also enhance how we report OldestXmin cutoff by putting it in context:
show how far behind it is at the _end_ of the VACUUM operation.

Deliberately don't do anything with VACUUM VERBOSE in this commit, since
a pending patch will generalize the log_autovacuum_min_duration code to
produce VACUUM VERBOSE output as well [1].  That'll get committed first.

[1] https://commitfest.postgresql.org/36/3431/
---
 src/include/commands/vacuum.h        |   2 +
 src/backend/access/heap/vacuumlazy.c | 108 +++++++++++++++++++--------
 src/backend/commands/analyze.c       |   3 +
 src/backend/commands/vacuum.c        |   9 +++
 4 files changed, 91 insertions(+), 31 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5a36049be..772a257fc 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -283,6 +283,8 @@ extern void vac_update_relstats(Relation relation,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
+								bool *frozenxid_updated,
+								bool *minmulti_updated,
 								bool in_outer_xact);
 extern void vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 148129e59..2950df1ce 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -197,6 +197,7 @@ typedef struct LVRelState
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
+	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
 
 	/* Statistics output by us, for table */
@@ -209,9 +210,10 @@ typedef struct LVRelState
 	int			num_index_scans;
 	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
+	int64		tuples_frozen;	/* # frozen by us */
 	int64		lpdead_items;	/* # deleted from indexes */
-	int64		new_dead_tuples;	/* new estimated total # of dead items in
-									 * table */
+	int64		recently_dead_tuples;	/* # dead, but not yet removable */
+	int64		missed_dead_tuples; /* # removable, but not removed */
 	int64		num_tuples;		/* total number of nonremovable tuples */
 	int64		live_tuples;	/* live tuples (reltuples estimate) */
 } LVRelState;
@@ -317,6 +319,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				write_rate;
 	bool		aggressive,
 				skipwithvm;
+	bool		frozenxid_updated,
+				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
@@ -535,9 +539,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
+		frozenxid_updated = minmulti_updated = false;
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							InvalidTransactionId, InvalidMultiXactId, false);
+							InvalidTransactionId, InvalidMultiXactId,
+							NULL, NULL, false);
 	}
 	else
 	{
@@ -546,7 +552,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff, false);
+							FreezeLimit, MultiXactCutoff,
+							&frozenxid_updated, &minmulti_updated, false);
 	}
 
 	/*
@@ -562,7 +569,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_report_vacuum(RelationGetRelid(rel),
 						 rel->rd_rel->relisshared,
 						 Max(new_live_tuples, 0),
-						 vacrel->new_dead_tuples);
+						 vacrel->recently_dead_tuples +
+						 vacrel->missed_dead_tuples);
 	pgstat_progress_end_command();
 
 	/* and log the action if appropriate */
@@ -576,6 +584,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
+			int32		diff;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -622,16 +631,41 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped using visibility map (%.2f%% of total)\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->frozenskipped_pages);
+							 orig_rel_pages - vacrel->scanned_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * (orig_rel_pages - vacrel->scanned_pages) / orig_rel_pages);
 			appendStringInfo(&buf,
-							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
+							 _("tuples: %lld removed, %lld remain (%lld newly frozen), %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
 							 (long long) vacrel->new_rel_tuples,
-							 (long long) vacrel->new_dead_tuples,
-							 OldestXmin);
+							 (long long) vacrel->tuples_frozen,
+							 (long long) vacrel->recently_dead_tuples);
+			if (vacrel->missed_dead_tuples > 0)
+				appendStringInfo(&buf,
+								 _("tuples missed: %lld dead from %u contended pages\n"),
+								 (long long) vacrel->missed_dead_tuples,
+								 vacrel->missed_dead_pages);
+			diff = (int32) (ReadNextTransactionId() - OldestXmin);
+			appendStringInfo(&buf,
+							 _("removable cutoff: %u, which is %d xids behind next\n"),
+							 OldestXmin, diff);
+			if (frozenxid_updated)
+			{
+				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				appendStringInfo(&buf,
+								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
+								 FreezeLimit, diff);
+			}
+			if (minmulti_updated)
+			{
+				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				appendStringInfo(&buf,
+								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
+								 MultiXactCutoff, diff);
+			}
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -787,13 +821,16 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	vacrel->frozenskipped_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
+	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
 
 	/* Initialize instrumentation counters */
 	vacrel->num_index_scans = 0;
 	vacrel->tuples_deleted = 0;
+	vacrel->tuples_frozen = 0;
 	vacrel->lpdead_items = 0;
-	vacrel->new_dead_tuples = 0;
+	vacrel->recently_dead_tuples = 0;
+	vacrel->missed_dead_tuples = 0;
 	vacrel->num_tuples = 0;
 	vacrel->live_tuples = 0;
 
@@ -1340,7 +1377,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	 * (unlikely) scenario that new_live_tuples is -1, take it as zero.
 	 */
 	vacrel->new_rel_tuples =
-		Max(vacrel->new_live_tuples, 0) + vacrel->new_dead_tuples;
+		Max(vacrel->new_live_tuples, 0) + vacrel->recently_dead_tuples +
+		vacrel->missed_dead_tuples;
 
 	/*
 	 * Release any remaining pin on visibility map page.
@@ -1405,7 +1443,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	initStringInfo(&buf);
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
-					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
+					 (long long) vacrel->recently_dead_tuples,
+					 vacrel->OldestXmin);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1586,7 +1625,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				lpdead_items,
-				new_dead_tuples,
+				recently_dead_tuples,
 				num_tuples,
 				live_tuples;
 	int			nnewlpdead;
@@ -1603,7 +1642,7 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	tuples_deleted = 0;
 	lpdead_items = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
 	num_tuples = 0;
 	live_tuples = 0;
 
@@ -1762,11 +1801,11 @@ retry:
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * If tuple is recently deleted then we must not remove it
-				 * from relation.  (We only remove items that are LP_DEAD from
+				 * If tuple is recently dead then we must not remove it from
+				 * the relation.  (We only remove items that are LP_DEAD from
 				 * pruning.)
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				prunestate->all_visible = false;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -1941,8 +1980,9 @@ retry:
 
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
+	vacrel->tuples_frozen += nfrozen;
 	vacrel->lpdead_items += lpdead_items;
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
 }
@@ -1995,7 +2035,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	int			lpdead_items,
 				num_tuples,
 				live_tuples,
-				new_dead_tuples;
+				recently_dead_tuples,
+				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 
@@ -2007,7 +2048,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	lpdead_items = 0;
 	num_tuples = 0;
 	live_tuples = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
+	missed_dead_tuples = 0;
 
 	maxoff = PageGetMaxOffsetNumber(page);
 	for (offnum = FirstOffsetNumber;
@@ -2081,16 +2123,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 				/*
 				 * There is some useful work for pruning to do, that won't be
 				 * done due to failure to get a cleanup lock.
-				 *
-				 * TODO Add dedicated instrumentation for this case
 				 */
+				missed_dead_tuples++;
 				break;
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 * Count in recently_dead_tuples, just like lazy_scan_prune
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
 
@@ -2118,13 +2159,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 		 *
 		 * We are not prepared to handle the corner case where a single pass
 		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
-		 * items.
+		 * items.  Count the LP_DEAD items as missed_dead_tuples instead. This
+		 * is slightly dishonest, but it's better than maintaining code to do
+		 * heap vacuuming for this one narrow corner case.
 		 */
 		if (lpdead_items > 0)
 			*hastup = true;
 		*hasfreespace = true;
 		num_tuples += lpdead_items;
-		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+		missed_dead_tuples += lpdead_items;
 	}
 	else if (lpdead_items > 0)
 	{
@@ -2159,9 +2202,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 	/*
 	 * Finally, add relevant page-local counts to whole-VACUUM counts
 	 */
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
+	vacrel->missed_dead_tuples += missed_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
+	if (missed_dead_tuples > 0)
+		vacrel->missed_dead_pages++;
 
 	/* Caller won't need to call lazy_scan_prune with same page */
 	return true;
@@ -2234,8 +2280,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * dead_items space is not CPU cache resident.
 		 *
 		 * We don't take any special steps to remember the LP_DEAD items (such
-		 * as counting them in new_dead_tuples report to the stats collector)
-		 * when the optimization is applied.  Though the accounting used in
+		 * as counting them in our final report to the stats collector) when
+		 * the optimization is applied.  Though the accounting used in
 		 * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD
 		 * items as dead rows in its own stats collector report, that's okay.
 		 * The discrepancy should be negligible.  If this optimization is ever
@@ -3375,7 +3421,7 @@ update_index_statistics(LVRelState *vacrel)
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
-							false);
+							NULL, NULL, false);
 	}
 }
 
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd77907fc..afd1cb8f5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -651,6 +651,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 
 		/* Same for indexes */
@@ -667,6 +668,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
+								NULL, NULL,
 								in_outer_xact);
 		}
 	}
@@ -679,6 +681,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		vac_update_relstats(onerel, -1, totalrows,
 							0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 	}
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index c94c187d3..d1d38d509 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1315,6 +1315,7 @@ vac_update_relstats(Relation relation,
 					BlockNumber num_all_visible_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
+					bool *frozenxid_updated, bool *minmulti_updated,
 					bool in_outer_xact)
 {
 	Oid			relid = RelationGetRelid(relation);
@@ -1390,22 +1391,30 @@ vac_update_relstats(Relation relation,
 	 * This should match vac_update_datfrozenxid() concerning what we consider
 	 * to be "in the future".
 	 */
+	if (frozenxid_updated)
+		*frozenxid_updated = false;
 	if (TransactionIdIsNormal(frozenxid) &&
 		pgcform->relfrozenxid != frozenxid &&
 		(TransactionIdPrecedes(pgcform->relfrozenxid, frozenxid) ||
 		 TransactionIdPrecedes(ReadNextTransactionId(),
 							   pgcform->relfrozenxid)))
 	{
+		if (frozenxid_updated)
+			*frozenxid_updated = true;
 		pgcform->relfrozenxid = frozenxid;
 		dirty = true;
 	}
 
 	/* Similarly for relminmxid */
+	if (minmulti_updated)
+		*minmulti_updated = false;
 	if (MultiXactIdIsValid(minmulti) &&
 		pgcform->relminmxid != minmulti &&
 		(MultiXactIdPrecedes(pgcform->relminmxid, minmulti) ||
 		 MultiXactIdPrecedes(ReadNextMultiXactId(), pgcform->relminmxid)))
 	{
+		if (minmulti_updated)
+			*minmulti_updated = true;
 		pgcform->relminmxid = minmulti;
 		dirty = true;
 	}
-- 
2.30.2

From be9b33339f0494ffd3b2327322433bacf72a5428 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 22 Nov 2021 10:02:30 -0800
Subject: [PATCH v5 4/5] Loosen coupling between relfrozenxid and tuple
 freezing.

Stop using tuple freezing (and MultiXact freezing) tuple header cutoffs
to determine the final relfrozenxid (and relminmxid) values that we set
for heap relations in pg_class.  Use "optimal" values instead.

Optimal values are the most recent values that are less than or equal to
any remaining XID/MultiXact in a tuple header (not counting frozen
xmin/xmax values).  This is now kept track of by VACUUM.  "Optimal"
values are always >= the tuple header FreezeLimit in an aggressive
VACUUM.  For a non-aggressive VACUUM, they can be less than or greater
than the tuple header FreezeLimit cutoff (though we still often pass
invalid values to indicate that we cannot advance relfrozenxid during
the VACUUM).
---
 src/include/access/heapam.h          |   4 +-
 src/include/access/heapam_xlog.h     |   4 +-
 src/include/commands/vacuum.h        |   1 +
 src/backend/access/heap/heapam.c     | 186 ++++++++++++++++++++-------
 src/backend/access/heap/vacuumlazy.c |  78 +++++++----
 src/backend/commands/cluster.c       |   5 +-
 src/backend/commands/vacuum.c        |  34 ++++-
 7 files changed, 231 insertions(+), 81 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index f3fb1e93a..bc5a96796 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -168,7 +168,9 @@ extern bool heap_freeze_tuple(HeapTupleHeader tuple,
 							  TransactionId relfrozenxid, TransactionId relminmxid,
 							  TransactionId cutoff_xid, TransactionId cutoff_multi);
 extern bool heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-									MultiXactId cutoff_multi, Buffer buf);
+									MultiXactId cutoff_multi,
+									TransactionId *NewRelfrozenxid,
+									MultiXactId *NewRelminmxid, Buffer buf);
 extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
 
 extern void simple_heap_insert(Relation relation, HeapTuple tup);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index ab9e873bc..b0ede623e 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -410,7 +410,9 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 									  TransactionId cutoff_xid,
 									  TransactionId cutoff_multi,
 									  xl_heap_freeze_tuple *frz,
-									  bool *totally_frozen);
+									  bool *totally_frozen,
+									  TransactionId *NewRelfrozenxid,
+									  MultiXactId *NewRelminmxid);
 extern void heap_execute_freeze_tuple(HeapTupleHeader tuple,
 									  xl_heap_freeze_tuple *xlrec_tp);
 extern XLogRecPtr log_heap_visible(RelFileNode rnode, Buffer heap_buffer,
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1848a65df..db813af99 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -291,6 +291,7 @@ extern bool vacuum_set_xid_limits(Relation rel,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
+								  MultiXactId *oldestMxact,
 								  TransactionId *freezeLimit,
 								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index a1bacb0eb..2f3399265 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6087,12 +6087,24 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
  * FRM_RETURN_IS_MULTI
  *		The return value is a new MultiXactId to set as new Xmax.
  *		(caller must obtain proper infomask bits using GetMultiXactIdHintBits)
+ *
+ * "NewRelfrozenxid" is an output value; it's used to maintain target new
+ * relfrozenxid for the relation.  It can be ignored unless "flags" contains
+ * either FRM_NOOP or FRM_RETURN_IS_MULTI, because we only handle multiXacts
+ * here.  This follows the general convention: only track XIDs that will still
+ * be in the table after the ongoing VACUUM finishes.  Note that it's up to
+ * caller to maintain this when the Xid return value is itself an Xid.
+ *
+ * Note that we cannot depend on xmin to maintain NewRelfrozenxid.  We need to
+ * push maintenance of NewRelfrozenxid down this far, since in general xmin
+ * might have been frozen by an earlier VACUUM operation, in which case our
+ * caller will not have factored-in xmin when maintaining NewRelfrozenxid.
  */
 static TransactionId
 FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 				  TransactionId relfrozenxid, TransactionId relminmxid,
 				  TransactionId cutoff_xid, MultiXactId cutoff_multi,
-				  uint16 *flags)
+				  uint16 *flags, TransactionId *NewRelfrozenxid)
 {
 	TransactionId xid = InvalidTransactionId;
 	int			i;
@@ -6104,6 +6116,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	bool		has_lockers;
 	TransactionId update_xid;
 	bool		update_committed;
+	TransactionId tempNewRelfrozenxid;
 
 	*flags = 0;
 
@@ -6198,13 +6211,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 
 	/* is there anything older than the cutoff? */
 	need_replace = false;
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	for (i = 0; i < nmembers; i++)
 	{
 		if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-		{
 			need_replace = true;
-			break;
-		}
+		if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+			tempNewRelfrozenxid = members[i].xid;
 	}
 
 	/*
@@ -6213,6 +6226,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 */
 	if (!need_replace)
 	{
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 		*flags |= FRM_NOOP;
 		pfree(members);
 		return InvalidTransactionId;
@@ -6222,6 +6236,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 * If the multi needs to be updated, figure out which members do we need
 	 * to keep.
 	 */
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	nnewmembers = 0;
 	newmembers = palloc(sizeof(MultiXactMember) * nmembers);
 	has_lockers = false;
@@ -6303,7 +6318,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			 * list.)
 			 */
 			if (TransactionIdIsValid(update_xid))
+			{
 				newmembers[nnewmembers++] = members[i];
+				if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+					tempNewRelfrozenxid = members[i].xid;
+			}
 		}
 		else
 		{
@@ -6313,6 +6332,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			{
 				/* running locker cannot possibly be older than the cutoff */
 				Assert(!TransactionIdPrecedes(members[i].xid, cutoff_xid));
+				Assert(!TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid));
 				newmembers[nnewmembers++] = members[i];
 				has_lockers = true;
 			}
@@ -6341,6 +6361,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		if (update_committed)
 			*flags |= FRM_MARK_COMMITTED;
 		xid = update_xid;
+		/* Caller manages NewRelfrozenxid directly when we return an XID */
 	}
 	else
 	{
@@ -6350,6 +6371,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		 */
 		xid = MultiXactIdCreateFromMembers(nnewmembers, newmembers);
 		*flags |= FRM_RETURN_IS_MULTI;
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 	}
 
 	pfree(newmembers);
@@ -6368,6 +6390,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
  * will be totally frozen after these operations are performed and false if
  * more freezing will eventually be required.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will actually go on to freeze as indicated by our *frz output, so
+ * any (xmin, xmax, xvac) XIDs that we indicate need to be frozen won't need
+ * to be counted here.  Values are valid lower bounds at the point that the
+ * ongoing VACUUM finishes.
+ *
  * Caller is responsible for setting the offset field, if appropriate.
  *
  * It is assumed that the caller has checked the tuple with
@@ -6392,7 +6421,9 @@ bool
 heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 						  TransactionId relfrozenxid, TransactionId relminmxid,
 						  TransactionId cutoff_xid, TransactionId cutoff_multi,
-						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p)
+						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p,
+						  TransactionId *NewRelfrozenxid,
+						  MultiXactId *NewRelminmxid)
 {
 	bool		changed = false;
 	bool		xmax_already_frozen = false;
@@ -6436,6 +6467,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->t_infomask |= HEAP_XMIN_FROZEN;
 			changed = true;
 		}
+		else if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+		{
+			/* won't be frozen, but older than current NewRelfrozenxid */
+			*NewRelfrozenxid = xid;
+		}
 	}
 
 	/*
@@ -6453,10 +6489,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 	{
 		TransactionId newxmax;
 		uint16		flags;
+		TransactionId temp = *NewRelfrozenxid;
 
 		newxmax = FreezeMultiXactId(xid, tuple->t_infomask,
 									relfrozenxid, relminmxid,
-									cutoff_xid, cutoff_multi, &flags);
+									cutoff_xid, cutoff_multi, &flags, &temp);
 
 		freeze_xmax = (flags & FRM_INVALIDATE_XMAX);
 
@@ -6474,6 +6511,24 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			if (flags & FRM_MARK_COMMITTED)
 				frz->t_infomask |= HEAP_XMAX_COMMITTED;
 			changed = true;
+
+			if (TransactionIdPrecedes(newxmax, *NewRelfrozenxid))
+			{
+				/* New xmax is an XID older than new NewRelfrozenxid */
+				*NewRelfrozenxid = newxmax;
+			}
+		}
+		else if (flags & FRM_NOOP)
+		{
+			/*
+			 * Changing nothing, so might have to ratchet back NewRelminmxid,
+			 * NewRelfrozenxid, or both together
+			 */
+			if (MultiXactIdIsValid(xid) &&
+				MultiXactIdPrecedes(xid, *NewRelminmxid))
+				*NewRelminmxid = xid;
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 		else if (flags & FRM_RETURN_IS_MULTI)
 		{
@@ -6495,6 +6550,13 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->xmax = newxmax;
 
 			changed = true;
+
+			/*
+			 * New multixact might have remaining XID older than
+			 * NewRelfrozenxid
+			 */
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 	}
 	else if (TransactionIdIsNormal(xid))
@@ -6522,7 +6584,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			freeze_xmax = true;
 		}
 		else
+		{
 			freeze_xmax = false;
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			{
+				/* won't be frozen, but older than current NewRelfrozenxid */
+				*NewRelfrozenxid = xid;
+			}
+		}
 	}
 	else if ((tuple->t_infomask & HEAP_XMAX_INVALID) ||
 			 !TransactionIdIsValid(HeapTupleHeaderGetRawXmax(tuple)))
@@ -6569,6 +6638,9 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 		 * was removed in PostgreSQL 9.0.  Note that if we were to respect
 		 * cutoff_xid here, we'd need to make surely to clear totally_frozen
 		 * when we skipped freezing on that basis.
+		 *
+		 * Since we always freeze here, NewRelfrozenxid doesn't need to be
+		 * maintained.
 		 */
 		if (TransactionIdIsNormal(xid))
 		{
@@ -6646,11 +6718,14 @@ heap_freeze_tuple(HeapTupleHeader tuple,
 	xl_heap_freeze_tuple frz;
 	bool		do_freeze;
 	bool		tuple_totally_frozen;
+	TransactionId NewRelfrozenxid = FirstNormalTransactionId;
+	MultiXactId NewRelminmxid = FirstMultiXactId;
 
 	do_freeze = heap_prepare_freeze_tuple(tuple,
 										  relfrozenxid, relminmxid,
 										  cutoff_xid, cutoff_multi,
-										  &frz, &tuple_totally_frozen);
+										  &frz, &tuple_totally_frozen,
+										  &NewRelfrozenxid, &NewRelminmxid);
 
 	/*
 	 * Note that because this is not a WAL-logged operation, we don't need to
@@ -7080,6 +7155,15 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  * Check to see whether any of the XID fields of a tuple (xmin, xmax, xvac)
  * are older than the specified cutoff XID or MultiXactId.  If so, return true.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will never freeze any of the XIDs from the tuple, even when we say
+ * that they should.  If caller opts to go with our recommendation to freeze,
+ * then it must account for the fact that it shouldn't trust how we've set
+ * NewRelfrozenxid/NewRelminmxid.  (In practice aggressive VACUUMs always take
+ * our recommendation because they must, and non-aggressive VACUUMs always opt
+ * to not freeze, preferring to ratchet back NewRelfrozenxid instead).
+ *
  * It doesn't matter whether the tuple is alive or dead, we are checking
  * to see if a tuple needs to be removed or frozen to avoid wraparound.
  *
@@ -7088,74 +7172,86 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  */
 bool
 heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-						MultiXactId cutoff_multi, Buffer buf)
+						MultiXactId cutoff_multi,
+						TransactionId *NewRelfrozenxid,
+						MultiXactId *NewRelminmxid, Buffer buf)
 {
 	TransactionId xid;
+	bool		needs_freeze = false;
 
 	xid = HeapTupleHeaderGetXmin(tuple);
-	if (TransactionIdIsNormal(xid) &&
-		TransactionIdPrecedes(xid, cutoff_xid))
-		return true;
+	if (TransactionIdIsNormal(xid))
+	{
+		if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			*NewRelfrozenxid = xid;
+		if (TransactionIdPrecedes(xid, cutoff_xid))
+			needs_freeze = true;
+	}
 
 	/*
 	 * The considerations for multixacts are complicated; look at
 	 * heap_prepare_freeze_tuple for justifications.  This routine had better
 	 * be in sync with that one!
+	 *
+	 * (Actually, we maintain NewRelminmxid differently here, because we
+	 * assume that XIDs that should be frozen according to cutoff_xid won't
+	 * be, whereas heap_prepare_freeze_tuple makes the opposite assumption.)
 	 */
 	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
 	{
 		MultiXactId multi;
+		MultiXactMember *members;
+		int			nmembers;
 
 		multi = HeapTupleHeaderGetRawXmax(tuple);
-		if (!MultiXactIdIsValid(multi))
-		{
-			/* no xmax set, ignore */
-			;
-		}
-		else if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
+		if (MultiXactIdIsValid(multi) &&
+			MultiXactIdPrecedes(multi, *NewRelminmxid))
+			*NewRelminmxid = multi;
+
+		if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
 			return true;
 		else if (MultiXactIdPrecedes(multi, cutoff_multi))
-			return true;
-		else
+			needs_freeze = true;
+
+		/* need to check whether any member of the mxact is too old */
+		nmembers = GetMultiXactIdMembers(multi, &members, false,
+										 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
+
+		for (int i = 0; i < nmembers; i++)
 		{
-			MultiXactMember *members;
-			int			nmembers;
-			int			i;
-
-			/* need to check whether any member of the mxact is too old */
-
-			nmembers = GetMultiXactIdMembers(multi, &members, false,
-											 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
-
-			for (i = 0; i < nmembers; i++)
-			{
-				if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-				{
-					pfree(members);
-					return true;
-				}
-			}
-			if (nmembers > 0)
-				pfree(members);
+			if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
+				needs_freeze = true;
+			if (TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
 		}
+		if (nmembers > 0)
+			pfree(members);
 	}
 	else
 	{
 		xid = HeapTupleHeaderGetRawXmax(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
 	if (tuple->t_infomask & HEAP_MOVED)
 	{
 		xid = HeapTupleHeaderGetXvac(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
-	return false;
+	return needs_freeze;
 }
 
 /*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 7614d6108..eade44ed0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -171,8 +171,10 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
-	/* Are FreezeLimit/MultiXactCutoff still valid? */
-	bool		freeze_cutoffs_valid;
+
+	/* Track new pg_class.relfrozenxid/pg_class.relminmxid values */
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -330,6 +332,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
 
@@ -366,8 +369,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 									   params->freeze_table_age,
 									   params->multixact_freeze_min_age,
 									   params->multixact_freeze_table_age,
-									   &OldestXmin, &FreezeLimit,
-									   &MultiXactCutoff);
+									   &OldestXmin, &OldestMxact,
+									   &FreezeLimit, &MultiXactCutoff);
 
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
@@ -432,8 +435,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
-	/* Track if cutoffs became invalid (possible in !aggressive case only) */
-	vacrel->freeze_cutoffs_valid = true;
+
+	/* Initialize values used to advance relfrozenxid/relminmxid at the end */
+	vacrel->NewRelfrozenxid = OldestXmin;
+	vacrel->NewRelminmxid = OldestMxact;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -522,16 +527,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
 	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
 	 * provided we didn't skip any all-visible (not all-frozen) pages using
-	 * the visibility map, and assuming that we didn't fail to get a cleanup
-	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
-	 * MultiXactCutoff) established for VACUUM operation.
+	 * the visibility map.  A non-aggressive VACUUM might only be able to
+	 * advance relfrozenxid to an XID from before FreezeLimit (or a relminmxid
+	 * from before MultiXactCutoff) when it wasn't possible to freeze some
+	 * tuples due to our inability to acquire a cleanup lock, but the effect
+	 * is usually insignificant -- NewRelfrozenxid value still has a decent
+	 * chance of being much more recent that the existing relfrozenxid.
 	 *
 	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
 	 * the rel_pages used by lazy_scan_heap, which won't match when we
 	 * happened to truncate the relation afterwards.
 	 */
-	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
-		!vacrel->freeze_cutoffs_valid)
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
@@ -548,7 +555,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff,
+							vacrel->NewRelfrozenxid, vacrel->NewRelminmxid,
 							&frozenxid_updated, &minmulti_updated, false);
 	}
 
@@ -650,17 +657,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 OldestXmin, diff);
 			if (frozenxid_updated)
 			{
-				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				diff = (int32) (vacrel->NewRelfrozenxid - vacrel->relfrozenxid);
 				appendStringInfo(&buf,
 								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
-								 FreezeLimit, diff);
+								 vacrel->NewRelfrozenxid, diff);
 			}
 			if (minmulti_updated)
 			{
-				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				diff = (int32) (vacrel->NewRelminmxid - vacrel->relminmxid);
 				appendStringInfo(&buf,
 								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
-								 MultiXactCutoff, diff);
+								 vacrel->NewRelminmxid, diff);
 			}
 			if (orig_rel_pages > 0)
 			{
@@ -1628,6 +1635,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	int			nfrozen;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1636,6 +1645,8 @@ lazy_scan_prune(LVRelState *vacrel,
 retry:
 
 	/* Initialize (or reset) page-level counters */
+	NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	NewRelminmxid = vacrel->NewRelminmxid;
 	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
@@ -1845,7 +1856,9 @@ retry:
 									  vacrel->FreezeLimit,
 									  vacrel->MultiXactCutoff,
 									  &frozen[nfrozen],
-									  &tuple_totally_frozen))
+									  &tuple_totally_frozen,
+									  &NewRelfrozenxid,
+									  &NewRelminmxid))
 		{
 			/* Will execute freeze below */
 			frozen[nfrozen++].offset = offnum;
@@ -1859,13 +1872,16 @@ retry:
 			prunestate->all_frozen = false;
 	}
 
+	vacrel->offnum = InvalidOffsetNumber;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
 	 * that remains and needs to be considered for freezing now (LP_UNUSED and
 	 * LP_REDIRECT items also remain, but are of no further interest to us).
 	 */
-	vacrel->offnum = InvalidOffsetNumber;
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
 
 	/*
 	 * Consider the need to freeze any items with tuple storage from the page
@@ -2009,9 +2025,9 @@ retry:
  * We'll always return true for a non-aggressive VACUUM, even when we know
  * that this will cause them to miss out on freezing tuples from before
  * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
- * lock.  This does mean that they definitely won't be able to advance
- * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
- * relminmxid).  Caller waits for full cleanup lock when we return false.
+ * lock.  This does mean that they will have NewRelfrozenxid ratcheting back
+ * to a known-safe value (same applies to NewRelminmxid).  Caller waits for
+ * full cleanup lock when we return false.
  *
  * See lazy_scan_prune for an explanation of hastup return flag.  The
  * hasfreespace flag instructs caller on whether or not it should do generic
@@ -2035,6 +2051,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	MultiXactId NewRelminmxid = vacrel->NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -2081,7 +2099,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
 		if (heap_tuple_needs_freeze(tupleheader,
 									vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
+									vacrel->MultiXactCutoff,
+									&NewRelfrozenxid, &NewRelminmxid, buf))
 		{
 			if (vacrel->aggressive)
 			{
@@ -2091,10 +2110,11 @@ lazy_scan_noprune(LVRelState *vacrel,
 			}
 
 			/*
-			 * Current non-aggressive VACUUM operation definitely won't be
-			 * able to advance relfrozenxid or relminmxid
+			 * A non-aggressive VACUUM doesn't have to wait on a cleanup lock
+			 * to ensure that it advances relfrozenxid to a sufficiently
+			 * recent XID that happens to be present on this page.  It can
+			 * just accept an older New/final relfrozenxid instead.
 			 */
-			vacrel->freeze_cutoffs_valid = false;
 		}
 
 		num_tuples++;
@@ -2144,6 +2164,14 @@ lazy_scan_noprune(LVRelState *vacrel,
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * We have committed to not freezing the tuples on this page (always
+	 * happens with a non-aggressive VACUUM), so make sure that the target
+	 * relfrozenxid/relminmxid values reflect the XIDs/MXIDs we encountered
+	 */
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
+
 	/*
 	 * Now save details of the LP_DEAD items from the page in vacrel (though
 	 * only when VACUUM uses two-pass strategy).
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 66b87347d..6bd6688ae 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -767,6 +767,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	TupleDesc	oldTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TupleDesc	newTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TransactionId OldestXmin;
+	MultiXactId oldestMxact;
 	TransactionId FreezeXid;
 	MultiXactId MultiXactCutoff;
 	bool		use_sort;
@@ -856,8 +857,8 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * Since we're going to rewrite the whole table anyway, there's no reason
 	 * not to be aggressive about this.
 	 */
-	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
+	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0, &OldestXmin, &oldestMxact,
+						  &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ddf6279c7..c39e8088a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -950,10 +950,28 @@ get_all_vacuum_rels(int options)
  * The output parameters are:
  * - oldestXmin is the Xid below which tuples deleted by any xact (that
  *   committed) should be considered DEAD, not just RECENTLY_DEAD.
- * - freezeLimit is the Xid below which all Xids are replaced by
- *	 FrozenTransactionId during vacuum.
+ * - oldestMxact is the Mxid below which MultiXacts are definitely not
+ *   seen as visible by any running transaction.
+ * - freezeLimit is the Xid below which all Xids are definitely replaced by
+ *   FrozenTransactionId during aggressive vacuums.
  * - multiXactCutoff is the value below which all MultiXactIds are removed
  *   from Xmax.
+ *
+ * oldestXmin and oldestMxact can be thought of as the most recent values that
+ * can ever be passed to vac_update_relstats() as frozenxid and minmulti
+ * arguments.  These exact values will be used when no newer XIDs or
+ * MultiXacts remain in the heap relation (e.g., with an empty table).  It's
+ * typical for vacuumlazy.c caller to notice that older XIDs/Multixacts remain
+ * in the table, which will force it to use older value.  These older final
+ * values may not be any newer than the preexisting frozenxid/minmulti values
+ * from pg_class in extreme cases.  The final values are frequently fairly
+ * close to the optimal values that we give to vacuumlazy.c, though.
+ *
+ * An aggressive VACUUM always provides vac_update_relstats() arguments that
+ * are >= freezeLimit and >= multiXactCutoff.  A non-aggressive VACUUM may
+ * provide arguments that are either newer or older than freezeLimit and
+ * multiXactCutoff, or non-valid values (indicating that pg_class level
+ * cutoffs cannot be advanced at all).
  */
 bool
 vacuum_set_xid_limits(Relation rel,
@@ -962,6 +980,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_min_age,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
+					  MultiXactId *oldestMxact,
 					  TransactionId *freezeLimit,
 					  MultiXactId *multiXactCutoff)
 {
@@ -970,7 +989,6 @@ vacuum_set_xid_limits(Relation rel,
 	int			effective_multixact_freeze_max_age;
 	TransactionId limit;
 	TransactionId safeLimit;
-	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
 	int			freezetable;
@@ -1066,9 +1084,11 @@ vacuum_set_xid_limits(Relation rel,
 						 effective_multixact_freeze_max_age / 2);
 	Assert(mxid_freezemin >= 0);
 
+	/* Remember for caller */
+	*oldestMxact = GetOldestMultiXactId();
+
 	/* compute the cutoff multi, being careful to generate a valid value */
-	oldestMxact = GetOldestMultiXactId();
-	mxactLimit = oldestMxact - mxid_freezemin;
+	mxactLimit = *oldestMxact - mxid_freezemin;
 	if (mxactLimit < FirstMultiXactId)
 		mxactLimit = FirstMultiXactId;
 
@@ -1083,8 +1103,8 @@ vacuum_set_xid_limits(Relation rel,
 				(errmsg("oldest multixact is far in the past"),
 				 errhint("Close open transactions with multixacts soon to avoid wraparound problems.")));
 		/* Use the safe limit, unless an older mxact is still running */
-		if (MultiXactIdPrecedes(oldestMxact, safeMxactLimit))
-			mxactLimit = oldestMxact;
+		if (MultiXactIdPrecedes(*oldestMxact, safeMxactLimit))
+			mxactLimit = *oldestMxact;
 		else
 			mxactLimit = safeMxactLimit;
 	}
-- 
2.30.2

From 6bbaf3b1a874e84a66b6dfa806c57a30f70f054a Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 13 Dec 2021 15:00:49 -0800
Subject: [PATCH v5 5/5] Freeze tuples early to advance relfrozenxid.

Freeze whenever pruning modified the page, or whenever we see that we're
going to mark the page all-visible without also marking it all-frozen.

There has been plenty of discussion of early/opportunistic freezing in
the past.  It is generally considered important as a way of minimizing
repeated dirtying of heap pages (or the total volume of FPIs in the WAL
stream) over time.  While that goal is certainly very important, this
patch has another priority: making VACUUM advance relfrozenxid sooner
and more frequently.

The overall effect is that tables like pgbench's history table can be
vacuumed very frequently, and have most individual vacuum operations
generate 0 FPIs in WAL -- they will never need an aggressive VACUUM.

GUCs like vacuum_freeze_min_age never made much sense after the freeze
map work in PostgreSQL 9.6.  The default is 50 million transactions,
which current tends to result in our being unable to freeze tuples
before the page is marked all-visible (but not all-frozen).  This
creates a huge performance cliff later on, during the first aggressive
VACUUM.  Freezing early effectively avoids accumulating "debt" from very
old unfrozen tuples.
---
 src/include/access/heapam.h          |  1 +
 src/backend/access/heap/pruneheap.c  |  8 ++-
 src/backend/access/heap/vacuumlazy.c | 87 +++++++++++++++++++++++++---
 3 files changed, 88 insertions(+), 8 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bc5a96796..9eaa365df 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -188,6 +188,7 @@ extern int	heap_page_prune(Relation relation, Buffer buffer,
 							struct GlobalVisState *vistest,
 							TransactionId old_snap_xmin,
 							TimestampTz old_snap_ts_ts,
+							bool *modified,
 							int	*nnewlpdead,
 							OffsetNumber *off_loc);
 extern void heap_page_prune_execute(Buffer buffer,
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 522a00af6..e95dea38d 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -182,11 +182,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
 		 */
 		if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
 		{
+			bool	modified;
 			int		ndeleted,
 					nnewlpdead;
 
 			ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
-									   limited_ts, &nnewlpdead, NULL);
+									   limited_ts, &modified, &nnewlpdead, NULL);
 
 			/*
 			 * Report the number of tuples reclaimed to pgstats.  This is
@@ -244,6 +245,7 @@ heap_page_prune(Relation relation, Buffer buffer,
 				GlobalVisState *vistest,
 				TransactionId old_snap_xmin,
 				TimestampTz old_snap_ts,
+				bool *modified,
 				int	*nnewlpdead,
 				OffsetNumber *off_loc)
 {
@@ -375,6 +377,8 @@ heap_page_prune(Relation relation, Buffer buffer,
 
 			PageSetLSN(BufferGetPage(buffer), recptr);
 		}
+
+		*modified = true;
 	}
 	else
 	{
@@ -387,12 +391,14 @@ heap_page_prune(Relation relation, Buffer buffer,
 		 * point in repeating the prune/defrag process until something else
 		 * happens to the page.
 		 */
+		*modified = false;
 		if (((PageHeader) page)->pd_prune_xid != prstate.new_prune_xid ||
 			PageIsFull(page))
 		{
 			((PageHeader) page)->pd_prune_xid = prstate.new_prune_xid;
 			PageClearFull(page);
 			MarkBufferDirtyHint(buffer, true);
+			*modified = true;
 		}
 	}
 
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index eade44ed0..a10faea6e 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -168,6 +168,7 @@ typedef struct LVRelState
 
 	/* VACUUM operation's cutoff for pruning */
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
@@ -358,11 +359,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/*
 	 * Get cutoffs that determine which tuples we need to freeze during the
-	 * VACUUM operation.
+	 * VACUUM operation.  This includes information that is used during
+	 * opportunistic freezing, where the most aggressive possible cutoffs
+	 * (OldestXmin and OldestMxact) are used for some heap pages, based on
+	 * considerations about cost.
 	 *
 	 * Also determines if this is to be an aggressive VACUUM.  This will
 	 * eventually be required for any table where (for whatever reason) no
 	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
+	 * This used to be much more common, but we now work hard to advance
+	 * relfrozenxid in non-aggressive VACUUMs.
 	 */
 	aggressive = vacuum_set_xid_limits(rel,
 									   params->freeze_min_age,
@@ -433,6 +439,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/* Set cutoffs for entire VACUUM */
 	vacrel->OldestXmin = OldestXmin;
+	vacrel->OldestMxact = OldestMxact;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
 
@@ -1637,6 +1644,10 @@ lazy_scan_prune(LVRelState *vacrel,
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 	TransactionId NewRelfrozenxid;
 	MultiXactId NewRelminmxid;
+	bool		modified;
+	TransactionId FreezeLimit = vacrel->FreezeLimit;
+	MultiXactId MultiXactCutoff = vacrel->MultiXactCutoff;
+	bool		earlyfreezing = false;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1663,8 +1674,19 @@ retry:
 	 * that were deleted from indexes.
 	 */
 	tuples_deleted = heap_page_prune(rel, buf, vistest,
-									 InvalidTransactionId, 0, &nnewlpdead,
-									 &vacrel->offnum);
+									 InvalidTransactionId, 0, &modified,
+									 &nnewlpdead, &vacrel->offnum);
+
+	/*
+	 * If page was modified during pruning, then perform early freezing
+	 * opportunistically
+	 */
+	if (!earlyfreezing && modified)
+	{
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+	}
 
 	/*
 	 * Now scan the page to collect LP_DEAD items and check for tuples
@@ -1719,7 +1741,7 @@ retry:
 		if (ItemIdIsDead(itemid))
 		{
 			deadoffsets[lpdead_items++] = offnum;
-			prunestate->all_visible = false;
+			/* Don't set all_visible to false just yet */
 			prunestate->has_lpdead_items = true;
 			continue;
 		}
@@ -1853,8 +1875,8 @@ retry:
 		if (heap_prepare_freeze_tuple(tuple.t_data,
 									  vacrel->relfrozenxid,
 									  vacrel->relminmxid,
-									  vacrel->FreezeLimit,
-									  vacrel->MultiXactCutoff,
+									  FreezeLimit,
+									  MultiXactCutoff,
 									  &frozen[nfrozen],
 									  &tuple_totally_frozen,
 									  &NewRelfrozenxid,
@@ -1874,6 +1896,57 @@ retry:
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * Reconsider applying early freezing before committing to processing the
+	 * page as currently planned.  There are 2 reasons to change our mind:
+	 *
+	 * 1. The standard FreezeLimit cutoff generally indicates that we should
+	 * freeze XIDs that are more than freeze_min_age XIDs in the past
+	 * (relative to OldestXmin).  But that should only be treated as a rough
+	 * guideline; it makes sense to freeze all eligible tuples on pages where
+	 * we're going to freeze at least one in any case.
+	 *
+	 * 2. If the page is now eligible to be marked all_visible, but is not
+	 * also eligible to be marked all_frozen, then we freeze early to make
+	 * sure that the page becomes all_frozen.  We should avoid building up
+	 * "freeze debt" that can only be paid off by an aggressive VACUUM, later
+	 * on.  This makes it much less likely that an aggressive VACUUM will ever
+	 * be required.
+	 *
+	 * Note: We deliberately track all_visible in a way that excludes LP_DEAD
+	 * items here.  Any page that is "all_visible for tuples with storage"
+	 * will be eligible to have its visibility map bit set during the ongoing
+	 * VACUUM, one way or another.  LP_DEAD items only make it unsafe to set
+	 * the page all_visible during the first heap pass, but the second heap
+	 * pass should be able to perform equivalent processing. (The second heap
+	 * pass cannot freeze tuples, though.)
+	 */
+	if (!earlyfreezing &&
+		((nfrozen > 0 && nfrozen < num_tuples) ||
+		 (prunestate->all_visible && !prunestate->all_frozen)))
+	{
+		/*
+		 * XXX Need to worry about leaking MultiXacts in FreezeMultiXactId()
+		 * now (via heap_prepare_freeze_tuple calls)?  That was already
+		 * possible, but presumably this makes it much more likely.
+		 *
+		 * On the other hand, that's only possible when we need to replace an
+		 * existing MultiXact with a new one.  Even then, we won't have
+		 * preallocated a new MultiXact (which we now risk leaking) if there
+		 * was only one remaining XID, and the XID is for an updater (we'll
+		 * only prepare to replace xmax with the XID directly).  So maybe it's
+		 * still a narrow enough problem to be ignored.
+		 */
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+		goto retry;
+	}
+
+	/* Time to define all_visible in a way that accounts for LP_DEAD items */
+	if (lpdead_items > 0)
+		prunestate->all_visible = false;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
@@ -1919,7 +1992,7 @@ retry:
 		{
 			XLogRecPtr	recptr;
 
-			recptr = log_heap_freeze(vacrel->rel, buf, vacrel->FreezeLimit,
+			recptr = log_heap_freeze(vacrel->rel, buf, FreezeLimit,
 									 frozen, nfrozen);
 			PageSetLSN(page, recptr);
 		}
-- 
2.30.2

From 13980cd0db4278d4d7c5ce9f9b048dff57019589 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 11 Dec 2021 17:39:45 -0800
Subject: [PATCH v5 3/5] Simplify vacuum_set_xid_limits signature.

Refactoring, making the return value of vacuum_set_xid_limits()
determine whether or not this will be an aggressive VACUUM.

This will make it easier to set/return an oldestMxact value for
vacuumlazy.c caller in the next commit, which is an important detail
that enables advancing relminmxid opportunistically.
---
 src/include/commands/vacuum.h        |   6 +-
 src/backend/access/heap/vacuumlazy.c |  32 +++----
 src/backend/commands/cluster.c       |   3 +-
 src/backend/commands/vacuum.c        | 134 +++++++++++++--------------
 4 files changed, 79 insertions(+), 96 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 772a257fc..1848a65df 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -286,15 +286,13 @@ extern void vac_update_relstats(Relation relation,
 								bool *frozenxid_updated,
 								bool *minmulti_updated,
 								bool in_outer_xact);
-extern void vacuum_set_xid_limits(Relation rel,
+extern bool vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
 								  TransactionId *freezeLimit,
-								  TransactionId *xidFullScanLimit,
-								  MultiXactId *multiXactCutoff,
-								  MultiXactId *mxactFullScanLimit);
+								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
 									  MultiXactId relminmxid);
 extern void vac_update_datfrozenxid(void);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2950df1ce..7614d6108 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -323,8 +323,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
-	TransactionId xidFullScanLimit;
-	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
@@ -355,24 +353,22 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
 								  RelationGetRelid(rel));
 
-	vacuum_set_xid_limits(rel,
-						  params->freeze_min_age,
-						  params->freeze_table_age,
-						  params->multixact_freeze_min_age,
-						  params->multixact_freeze_table_age,
-						  &OldestXmin, &FreezeLimit, &xidFullScanLimit,
-						  &MultiXactCutoff, &mxactFullScanLimit);
-
 	/*
-	 * We request an aggressive scan if the table's frozen Xid is now older
-	 * than or equal to the requested Xid full-table scan limit; or if the
-	 * table's minimum MultiXactId is older than or equal to the requested
-	 * mxid full-table scan limit; or if DISABLE_PAGE_SKIPPING was specified.
+	 * Get cutoffs that determine which tuples we need to freeze during the
+	 * VACUUM operation.
+	 *
+	 * Also determines if this is to be an aggressive VACUUM.  This will
+	 * eventually be required for any table where (for whatever reason) no
+	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
 	 */
-	aggressive = TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
-											   xidFullScanLimit);
-	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
-											  mxactFullScanLimit);
+	aggressive = vacuum_set_xid_limits(rel,
+									   params->freeze_min_age,
+									   params->freeze_table_age,
+									   params->multixact_freeze_min_age,
+									   params->multixact_freeze_table_age,
+									   &OldestXmin, &FreezeLimit,
+									   &MultiXactCutoff);
+
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
 	{
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 9d22f648a..66b87347d 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -857,8 +857,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * not to be aggressive about this.
 	 */
 	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, NULL, &MultiXactCutoff,
-						  NULL);
+						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index d1d38d509..ddf6279c7 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -942,25 +942,20 @@ get_all_vacuum_rels(int options)
  *
  * Input parameters are the target relation, applicable freeze age settings.
  *
+ * Return value indicates whether caller should do an aggressive VACUUM or
+ * not.  This is a VACUUM that cannot skip any pages using the visibility map
+ * (except all-frozen pages), which is guaranteed to be able to advance
+ * relfrozenxid and relminmxid.
+ *
  * The output parameters are:
- * - oldestXmin is the cutoff value used to distinguish whether tuples are
- *	 DEAD or RECENTLY_DEAD (see HeapTupleSatisfiesVacuum).
+ * - oldestXmin is the Xid below which tuples deleted by any xact (that
+ *   committed) should be considered DEAD, not just RECENTLY_DEAD.
  * - freezeLimit is the Xid below which all Xids are replaced by
  *	 FrozenTransactionId during vacuum.
- * - xidFullScanLimit (computed from freeze_table_age parameter)
- *	 represents a minimum Xid value; a table whose relfrozenxid is older than
- *	 this will have a full-table vacuum applied to it, to freeze tuples across
- *	 the whole table.  Vacuuming a table younger than this value can use a
- *	 partial scan.
- * - multiXactCutoff is the value below which all MultiXactIds are removed from
- *	 Xmax.
- * - mxactFullScanLimit is a value against which a table's relminmxid value is
- *	 compared to produce a full-table vacuum, as with xidFullScanLimit.
- *
- * xidFullScanLimit and mxactFullScanLimit can be passed as NULL if caller is
- * not interested.
+ * - multiXactCutoff is the value below which all MultiXactIds are removed
+ *   from Xmax.
  */
-void
+bool
 vacuum_set_xid_limits(Relation rel,
 					  int freeze_min_age,
 					  int freeze_table_age,
@@ -968,9 +963,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
 					  TransactionId *freezeLimit,
-					  TransactionId *xidFullScanLimit,
-					  MultiXactId *multiXactCutoff,
-					  MultiXactId *mxactFullScanLimit)
+					  MultiXactId *multiXactCutoff)
 {
 	int			freezemin;
 	int			mxid_freezemin;
@@ -980,6 +973,7 @@ vacuum_set_xid_limits(Relation rel,
 	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
+	int			freezetable;
 
 	/*
 	 * We can always ignore processes running lazy vacuum.  This is because we
@@ -1097,64 +1091,60 @@ vacuum_set_xid_limits(Relation rel,
 
 	*multiXactCutoff = mxactLimit;
 
-	if (xidFullScanLimit != NULL)
-	{
-		int			freezetable;
+	/*
+	 * Done setting output parameters; just need to figure out if caller needs
+	 * to do an aggressive VACUUM or not.
+	 *
+	 * Determine the table freeze age to use: as specified by the caller, or
+	 * vacuum_freeze_table_age, but in any case not more than
+	 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
+	 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
+	 * before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_freeze_table_age;
+	freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		Assert(mxactFullScanLimit != NULL);
+	/*
+	 * Compute XID limit causing an aggressive vacuum, being careful not to
+	 * generate a "permanent" XID
+	 */
+	limit = ReadNextTransactionId() - freezetable;
+	if (!TransactionIdIsNormal(limit))
+		limit = FirstNormalTransactionId;
+	if (TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
+									  limit))
+		return true;
 
-		/*
-		 * Determine the table freeze age to use: as specified by the caller,
-		 * or vacuum_freeze_table_age, but in any case not more than
-		 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
-		 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
-		 * before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_freeze_table_age;
-		freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
+	/*
+	 * Similar to the above, determine the table freeze age to use for
+	 * multixacts: as specified by the caller, or
+	 * vacuum_multixact_freeze_table_age, but in any case not more than
+	 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have e.g.
+	 * nightly VACUUM schedule, the nightly VACUUM gets a chance to freeze
+	 * multixacts before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = multixact_freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_multixact_freeze_table_age;
+	freezetable = Min(freezetable,
+					  effective_multixact_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		/*
-		 * Compute XID limit causing a full-table vacuum, being careful not to
-		 * generate a "permanent" XID.
-		 */
-		limit = ReadNextTransactionId() - freezetable;
-		if (!TransactionIdIsNormal(limit))
-			limit = FirstNormalTransactionId;
+	/*
+	 * Compute MultiXact limit causing an aggressive vacuum, being careful to
+	 * generate a valid MultiXact value
+	 */
+	mxactLimit = ReadNextMultiXactId() - freezetable;
+	if (mxactLimit < FirstMultiXactId)
+		mxactLimit = FirstMultiXactId;
+	if (MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
+									mxactLimit))
+		return true;
 
-		*xidFullScanLimit = limit;
-
-		/*
-		 * Similar to the above, determine the table freeze age to use for
-		 * multixacts: as specified by the caller, or
-		 * vacuum_multixact_freeze_table_age, but in any case not more than
-		 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have
-		 * e.g. nightly VACUUM schedule, the nightly VACUUM gets a chance to
-		 * freeze multixacts before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = multixact_freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_multixact_freeze_table_age;
-		freezetable = Min(freezetable,
-						  effective_multixact_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
-
-		/*
-		 * Compute MultiXact limit causing a full-table vacuum, being careful
-		 * to generate a valid MultiXact value.
-		 */
-		mxactLimit = ReadNextMultiXactId() - freezetable;
-		if (mxactLimit < FirstMultiXactId)
-			mxactLimit = FirstMultiXactId;
-
-		*mxactFullScanLimit = mxactLimit;
-	}
-	else
-	{
-		Assert(mxactFullScanLimit == NULL);
-	}
+	return false;
 }
 
 /*
-- 
2.30.2

From 6345b080a521277746a0088980a3a44dce66367d Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v5 1/5] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

We now also collect LP_DEAD items in the dead_items array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

We no longer report on "pin skipped pages" in log output.  A later patch
will add back an improved version of the same instrumentation.  We don't
want to show any information about any failures to acquire cleanup locks
unless we actually failed to do useful work as a consequence.  A page
that we could not acquire a cleanup lock on is now treated as equivalent
to any other scanned page in most cases.
---
 src/backend/access/heap/vacuumlazy.c          | 815 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 518 insertions(+), 306 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index cd603e6aa..148129e59 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -143,6 +143,10 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
+	/* Use visibility map to skip? (disabled via reloption) */
+	bool		skipwithvm;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -167,6 +171,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -187,10 +193,8 @@ typedef struct LVRelState
 	 */
 	VacDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -203,6 +207,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -242,19 +247,22 @@ static int	elevel = -1;
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup, bool *hasfreespace);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void lazy_cleanup_all_indexes(LVRelState *vacrel);
 static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -307,16 +315,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive,
+				skipwithvm;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -362,8 +369,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 											   xidFullScanLimit);
 	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
 											  mxactFullScanLimit);
+	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
+	{
+		/*
+		 * Force aggressive mode, and disable skipping blocks using the
+		 * visibility map (even those set all-frozen)
+		 */
 		aggressive = true;
+		skipwithvm = false;
+	}
 
 	vacrel = (LVRelState *) palloc0(sizeof(LVRelState));
 
@@ -371,6 +386,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->rel = rel;
 	vac_open_indexes(vacrel->rel, RowExclusiveLock, &vacrel->nindexes,
 					 &vacrel->indrels);
+	vacrel->aggressive = aggressive;
+	vacrel->skipwithvm = skipwithvm;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 
@@ -415,6 +432,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	vacrel->relnamespace = get_namespace_name(RelationGetNamespace(rel));
 	vacrel->relname = pstrdup(RelationGetRelationName(rel));
@@ -451,30 +470,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, params->nworkers);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		/*
@@ -505,28 +510,44 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
+	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
+	 * provided we didn't skip any all-visible (not all-frozen) pages using
+	 * the visibility map, and assuming that we didn't fail to get a cleanup
+	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
+	 * MultiXactCutoff) established for VACUUM operation.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_heap, which won't match when we
+	 * happened to truncate the relation afterwards.
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -555,7 +576,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -602,10 +622,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -613,7 +632,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->pages_removed;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -730,7 +748,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, int nworkers)
 {
 	VacDeadItems *dead_items;
 	BlockNumber nblocks,
@@ -752,7 +770,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	pg_rusage_init(&ru0);
 
-	if (aggressive)
+	if (vacrel->aggressive)
 		ereport(elevel,
 				(errmsg("aggressively vacuuming \"%s.%s\"",
 						vacrel->relnamespace,
@@ -764,14 +782,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 						vacrel->relname)));
 
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
-	next_unskippable_block = 0;
-	next_failsafe_block = 0;
-	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -795,14 +808,16 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * dangerously old.
 	 */
 	lazy_check_wraparound_failsafe(vacrel);
+	next_failsafe_block = 0;
 
 	/*
 	 * Allocate the space for dead_items.  Note that this handles parallel
 	 * VACUUM initialization as part of allocating shared memory space used
 	 * for dead_items.
 	 */
-	dead_items_alloc(vacrel, params->nworkers);
+	dead_items_alloc(vacrel, nworkers);
 	dead_items = vacrel->dead_items;
+	next_fsm_block_to_vacuum = 0;
 
 	/* Report that we're scanning the heap, advertising total # of blocks */
 	initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
@@ -811,7 +826,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/*
-	 * Except when aggressive is set, we want to skip pages that are
+	 * Set things up for skipping blocks using visibility map.
+	 *
+	 * Except when vacrel->aggressive is set, we want to skip pages that are
 	 * all-visible according to the visibility map, but only when we can skip
 	 * at least SKIP_PAGES_THRESHOLD consecutive pages.  Since we're reading
 	 * sequentially, the OS should be doing readahead for us, so there's no
@@ -820,8 +837,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * page means that we can't update relfrozenxid, so we only want to do it
 	 * if we can skip a goodly number of pages.
 	 *
-	 * When aggressive is set, we can't skip pages just because they are
-	 * all-visible, but we can still skip pages that are all-frozen, since
+	 * When vacrel->aggressive is set, we can't skip pages just because they
+	 * are all-visible, but we can still skip pages that are all-frozen, since
 	 * such pages do not need freezing and do not affect the value that we can
 	 * safely set for relfrozenxid or relminmxid.
 	 *
@@ -844,17 +861,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
-	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+	next_unskippable_block = 0;
+	if (vacrel->skipwithvm)
 	{
 		while (next_unskippable_block < nblocks)
 		{
@@ -863,7 +872,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vmstatus = visibilitymap_get_status(vacrel->rel,
 												next_unskippable_block,
 												&vmbuffer);
-			if (aggressive)
+			if (vacrel->aggressive)
 			{
 				if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
 					break;
@@ -890,13 +899,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
@@ -906,7 +908,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		{
 			/* Time to advance next_unskippable_block */
 			next_unskippable_block++;
-			if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+			if (vacrel->skipwithvm)
 			{
 				while (next_unskippable_block < nblocks)
 				{
@@ -915,7 +917,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 					vmskipflags = visibilitymap_get_status(vacrel->rel,
 														   next_unskippable_block,
 														   &vmbuffer);
-					if (aggressive)
+					if (vacrel->aggressive)
 					{
 						if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
 							break;
@@ -944,19 +946,25 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			 * it's not all-visible.  But in an aggressive vacuum we know only
 			 * that it's not all-frozen, so it might still be all-visible.
 			 */
-			if (aggressive && VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
+			if (vacrel->aggressive &&
+				VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
 				all_visible_according_to_vm = true;
 		}
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current page can be skipped if we've seen a long enough run
+			 * of skippable blocks to justify skipping it -- provided it's not
+			 * the last page in the relation (according to rel_pages/nblocks).
+			 *
+			 * We always scan the table's last page to determine whether it
+			 * has tuples or not, even if it would otherwise be skipped
+			 * (unless we're skipping every single page in the relation). This
+			 * avoids having lazy_truncate_heap() take access-exclusive lock
+			 * on the table to attempt a truncation that just fails
+			 * immediately because there are tuples on the last page.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks && blkno < nblocks - 1)
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -965,18 +973,32 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * careful to count it as a skipped all-frozen page in that
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
-				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * know whether it was initially all-frozen, so we have to
+				 * recheck.
 				 */
-				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+				if (vacrel->aggressive ||
+					VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise it must be an all-visible (and possibly even
+			 * all-frozen) page that we decided to process regardless
+			 * (SKIP_PAGES_THRESHOLD must not have been crossed).
+			 */
 			all_visible_according_to_vm = true;
 		}
 
 		vacuum_delay_point();
 
+		/*
+		 * We're not skipping this page using the visibility map, and so it is
+		 * (by definition) a scanned page.  Any tuples from this page are now
+		 * guaranteed to be counted below, after some preparatory checks.
+		 */
+		vacrel->scanned_pages++;
+
 		/*
 		 * Regularly check if wraparound failsafe should trigger.
 		 *
@@ -1031,174 +1053,78 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
-		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
-		 * already have the correct page pinned anyway.  However, it's
-		 * possible that (a) next_unskippable_block is covered by a different
-		 * VM page than the current block or (b) we released our pin and did a
-		 * cycle of index vacuuming.
+		 * already have the correct page pinned anyway.
 		 */
 		visibilitymap_pin(vacrel->rel, blkno, &vmbuffer);
 
+		/* Finished preparatory checks.  Actually scan the page. */
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing using lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
-			bool		hastup;
+			bool		hastup,
+						hasfreespace;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
-			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				continue;
-			}
-
-			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
-			 */
 			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
+				/* Processed as new/empty page (lock and pin released) */
 				continue;
 			}
-			if (!aggressive)
+
+			/* Collect LP_DEAD items in dead_items array, count tuples */
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
+								  &hasfreespace))
 			{
+				Size		freespace;
+
 				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
+				 * Processed page successfully (without cleanup lock) -- just
+				 * need to perform rel truncation and FSM steps, much like the
+				 * lazy_scan_prune case.  Don't bother trying to match its
+				 * visibility map setting steps, though.
 				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
 				if (hastup)
 					vacrel->nonempty_pages = blkno + 1;
+				if (hasfreespace)
+					freespace = PageGetHeapFreeSpace(page);
+				UnlockReleaseBuffer(buf);
+				if (hasfreespace)
+					RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
 			}
+
+			/*
+			 * lazy_scan_noprune could not do all required processing.  Wait
+			 * for a cleanup lock, and call lazy_scan_prune in the usual way.
+			 */
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Processed as new/empty page (lock and pin released) */
 			continue;
 		}
 
 		/*
-		 * Prune and freeze tuples.
+		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
 		 * dead_items array.  This includes LP_DEAD line pointers that we
@@ -1406,7 +1332,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1480,14 +1406,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	appendStringInfo(&buf,
 					 _("%lld dead row versions cannot be removed yet, oldest xmin: %u\n"),
 					 (long long) vacrel->new_dead_tuples, vacrel->OldestXmin);
-	appendStringInfo(&buf, ngettext("Skipped %u page due to buffer pins, ",
-									"Skipped %u pages due to buffer pins, ",
-									vacrel->pinskipped_pages),
-					 vacrel->pinskipped_pages);
-	appendStringInfo(&buf, ngettext("%u frozen page.\n",
-									"%u frozen pages.\n",
-									vacrel->frozenskipped_pages),
-					 vacrel->frozenskipped_pages);
 	appendStringInfo(&buf, _("%s."), pg_rusage_show(&ru0));
 
 	ereport(elevel,
@@ -1501,6 +1419,137 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pfree(buf.data);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller must hold at least a shared lock.  We might need to escalate the
+ * lock in that case, so the type of lock caller holds needs to be specified
+ * using 'sharelock' argument.
+ *
+ * Returns false in common case where caller should go on to call
+ * lazy_scan_prune (or lazy_scan_noprune).  Otherwise returns true, indicating
+ * that lazy_scan_heap is done processing the page, releasing lock on caller's
+ * behalf.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never discover the space on a promoted standby.
+		 * The harm of repeated checking ought to normally not be too bad. The
+		 * space usually should be used at some point, otherwise there
+		 * wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1545,6 +1594,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
 	maxoff = PageGetMaxOffsetNumber(page);
 
 retry:
@@ -1607,10 +1658,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -1897,6 +1947,226 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() variant without pruning
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED in lazy_vacuum_heap_page.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).
+ *
+ * Returns true to indicate that all required processing has been performed.
+ * We'll always return true for a non-aggressive VACUUM, even when we know
+ * that this will cause them to miss out on freezing tuples from before
+ * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
+ * lock.  This does mean that they definitely won't be able to advance
+ * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
+ * relminmxid).  Caller waits for full cleanup lock when we return false.
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.  The
+ * hasfreespace flag instructs caller on whether or not it should do generic
+ * FSM processing for page, which is determined based on almost the same
+ * criteria as the lazy_scan_prune case.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup,
+				  bool *hasfreespace)
+{
+	OffsetNumber offnum,
+				maxoff;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
+	*hastup = false;			/* for now */
+	*hasfreespace = false;		/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		*hastup = true;		/* page prevents rel truncation */
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (heap_tuple_needs_freeze(tupleheader,
+									vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			/*
+			 * Current non-aggressive VACUUM operation definitely won't be
+			 * able to advance relfrozenxid or relminmxid
+			 */
+			vacrel->freeze_cutoffs_valid = false;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+
+				/*
+				 * There is some useful work for pruning to do, that won't be
+				 * done due to failure to get a cleanup lock.
+				 *
+				 * TODO Add dedicated instrumentation for this case
+				 */
+				break;
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in vacrel (though
+	 * only when VACUUM uses two-pass strategy).
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/*
+		 * Using one-pass strategy.
+		 *
+		 * We are not prepared to handle the corner case where a single pass
+		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
+		 * items.
+		 */
+		if (lpdead_items > 0)
+			*hastup = true;
+		*hasfreespace = true;
+		num_tuples += lpdead_items;
+		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+	}
+	else if (lpdead_items > 0)
+	{
+		VacDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * Caller won't be vacuuming this page later, so tell it to record
+		 * page's freespace in the FSM now
+		 */
+		*hasfreespace = true;
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Remove the collected garbage tuples from the table and its indexes.
  *
@@ -2342,67 +2612,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2468,7 +2677,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -2485,7 +2694,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 		/* Outsource everything to parallel variant */
 		parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
 											vacrel->num_index_scans,
-											(vacrel->tupcount_pages < vacrel->rel_pages));
+											(vacrel->scanned_pages < vacrel->rel_pages));
 	}
 }
 
@@ -2592,7 +2801,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From 17a28a0ea04b9b6f1e0b7177b13fe931874e5dca Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 22 Nov 2021 10:02:30 -0800
Subject: [PATCH v6 4/5] Loosen coupling between relfrozenxid and tuple
 freezing.

Stop using tuple freezing (and MultiXact freezing) tuple header cutoffs
to determine the final relfrozenxid (and relminmxid) values that we set
for heap relations in pg_class.  Use "optimal" values instead.

Optimal values are the most recent values that are less than or equal to
any remaining XID/MultiXact in a tuple header (not counting frozen
xmin/xmax values).  This is now kept track of by VACUUM.  "Optimal"
values are always >= the tuple header FreezeLimit in an aggressive
VACUUM.  For a non-aggressive VACUUM, they can be less than or greater
than the tuple header FreezeLimit cutoff (though we still often pass
invalid values to indicate that we cannot advance relfrozenxid during
the VACUUM).
---
 src/include/access/heapam.h          |   4 +-
 src/include/access/heapam_xlog.h     |   4 +-
 src/include/commands/vacuum.h        |   1 +
 src/backend/access/heap/heapam.c     | 186 ++++++++++++++++++++-------
 src/backend/access/heap/vacuumlazy.c |  78 +++++++----
 src/backend/commands/cluster.c       |   5 +-
 src/backend/commands/vacuum.c        |  34 ++++-
 7 files changed, 231 insertions(+), 81 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 0ad87730e..d35402f9f 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -168,7 +168,9 @@ extern bool heap_freeze_tuple(HeapTupleHeader tuple,
 							  TransactionId relfrozenxid, TransactionId relminmxid,
 							  TransactionId cutoff_xid, TransactionId cutoff_multi);
 extern bool heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-									MultiXactId cutoff_multi, Buffer buf);
+									MultiXactId cutoff_multi,
+									TransactionId *NewRelfrozenxid,
+									MultiXactId *NewRelminmxid, Buffer buf);
 extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
 
 extern void simple_heap_insert(Relation relation, HeapTuple tup);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 5c47fdcec..ae55c90f7 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -410,7 +410,9 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 									  TransactionId cutoff_xid,
 									  TransactionId cutoff_multi,
 									  xl_heap_freeze_tuple *frz,
-									  bool *totally_frozen);
+									  bool *totally_frozen,
+									  TransactionId *NewRelfrozenxid,
+									  MultiXactId *NewRelminmxid);
 extern void heap_execute_freeze_tuple(HeapTupleHeader tuple,
 									  xl_heap_freeze_tuple *xlrec_tp);
 extern XLogRecPtr log_heap_visible(RelFileNode rnode, Buffer heap_buffer,
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f..ead88edda 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -291,6 +291,7 @@ extern bool vacuum_set_xid_limits(Relation rel,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
+								  MultiXactId *oldestMxact,
 								  TransactionId *freezeLimit,
 								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 6ec57f3d8..521eb8044 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6087,12 +6087,24 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
  * FRM_RETURN_IS_MULTI
  *		The return value is a new MultiXactId to set as new Xmax.
  *		(caller must obtain proper infomask bits using GetMultiXactIdHintBits)
+ *
+ * "NewRelfrozenxid" is an output value; it's used to maintain target new
+ * relfrozenxid for the relation.  It can be ignored unless "flags" contains
+ * either FRM_NOOP or FRM_RETURN_IS_MULTI, because we only handle multiXacts
+ * here.  This follows the general convention: only track XIDs that will still
+ * be in the table after the ongoing VACUUM finishes.  Note that it's up to
+ * caller to maintain this when the Xid return value is itself an Xid.
+ *
+ * Note that we cannot depend on xmin to maintain NewRelfrozenxid.  We need to
+ * push maintenance of NewRelfrozenxid down this far, since in general xmin
+ * might have been frozen by an earlier VACUUM operation, in which case our
+ * caller will not have factored-in xmin when maintaining NewRelfrozenxid.
  */
 static TransactionId
 FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 				  TransactionId relfrozenxid, TransactionId relminmxid,
 				  TransactionId cutoff_xid, MultiXactId cutoff_multi,
-				  uint16 *flags)
+				  uint16 *flags, TransactionId *NewRelfrozenxid)
 {
 	TransactionId xid = InvalidTransactionId;
 	int			i;
@@ -6104,6 +6116,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	bool		has_lockers;
 	TransactionId update_xid;
 	bool		update_committed;
+	TransactionId tempNewRelfrozenxid;
 
 	*flags = 0;
 
@@ -6198,13 +6211,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 
 	/* is there anything older than the cutoff? */
 	need_replace = false;
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	for (i = 0; i < nmembers; i++)
 	{
 		if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-		{
 			need_replace = true;
-			break;
-		}
+		if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+			tempNewRelfrozenxid = members[i].xid;
 	}
 
 	/*
@@ -6213,6 +6226,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 */
 	if (!need_replace)
 	{
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 		*flags |= FRM_NOOP;
 		pfree(members);
 		return InvalidTransactionId;
@@ -6222,6 +6236,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 * If the multi needs to be updated, figure out which members do we need
 	 * to keep.
 	 */
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	nnewmembers = 0;
 	newmembers = palloc(sizeof(MultiXactMember) * nmembers);
 	has_lockers = false;
@@ -6303,7 +6318,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			 * list.)
 			 */
 			if (TransactionIdIsValid(update_xid))
+			{
 				newmembers[nnewmembers++] = members[i];
+				if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+					tempNewRelfrozenxid = members[i].xid;
+			}
 		}
 		else
 		{
@@ -6313,6 +6332,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			{
 				/* running locker cannot possibly be older than the cutoff */
 				Assert(!TransactionIdPrecedes(members[i].xid, cutoff_xid));
+				Assert(!TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid));
 				newmembers[nnewmembers++] = members[i];
 				has_lockers = true;
 			}
@@ -6341,6 +6361,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		if (update_committed)
 			*flags |= FRM_MARK_COMMITTED;
 		xid = update_xid;
+		/* Caller manages NewRelfrozenxid directly when we return an XID */
 	}
 	else
 	{
@@ -6350,6 +6371,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		 */
 		xid = MultiXactIdCreateFromMembers(nnewmembers, newmembers);
 		*flags |= FRM_RETURN_IS_MULTI;
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 	}
 
 	pfree(newmembers);
@@ -6368,6 +6390,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
  * will be totally frozen after these operations are performed and false if
  * more freezing will eventually be required.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will actually go on to freeze as indicated by our *frz output, so
+ * any (xmin, xmax, xvac) XIDs that we indicate need to be frozen won't need
+ * to be counted here.  Values are valid lower bounds at the point that the
+ * ongoing VACUUM finishes.
+ *
  * Caller is responsible for setting the offset field, if appropriate.
  *
  * It is assumed that the caller has checked the tuple with
@@ -6392,7 +6421,9 @@ bool
 heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 						  TransactionId relfrozenxid, TransactionId relminmxid,
 						  TransactionId cutoff_xid, TransactionId cutoff_multi,
-						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p)
+						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p,
+						  TransactionId *NewRelfrozenxid,
+						  MultiXactId *NewRelminmxid)
 {
 	bool		changed = false;
 	bool		xmax_already_frozen = false;
@@ -6436,6 +6467,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->t_infomask |= HEAP_XMIN_FROZEN;
 			changed = true;
 		}
+		else if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+		{
+			/* won't be frozen, but older than current NewRelfrozenxid */
+			*NewRelfrozenxid = xid;
+		}
 	}
 
 	/*
@@ -6453,10 +6489,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 	{
 		TransactionId newxmax;
 		uint16		flags;
+		TransactionId temp = *NewRelfrozenxid;
 
 		newxmax = FreezeMultiXactId(xid, tuple->t_infomask,
 									relfrozenxid, relminmxid,
-									cutoff_xid, cutoff_multi, &flags);
+									cutoff_xid, cutoff_multi, &flags, &temp);
 
 		freeze_xmax = (flags & FRM_INVALIDATE_XMAX);
 
@@ -6474,6 +6511,24 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			if (flags & FRM_MARK_COMMITTED)
 				frz->t_infomask |= HEAP_XMAX_COMMITTED;
 			changed = true;
+
+			if (TransactionIdPrecedes(newxmax, *NewRelfrozenxid))
+			{
+				/* New xmax is an XID older than new NewRelfrozenxid */
+				*NewRelfrozenxid = newxmax;
+			}
+		}
+		else if (flags & FRM_NOOP)
+		{
+			/*
+			 * Changing nothing, so might have to ratchet back NewRelminmxid,
+			 * NewRelfrozenxid, or both together
+			 */
+			if (MultiXactIdIsValid(xid) &&
+				MultiXactIdPrecedes(xid, *NewRelminmxid))
+				*NewRelminmxid = xid;
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 		else if (flags & FRM_RETURN_IS_MULTI)
 		{
@@ -6495,6 +6550,13 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->xmax = newxmax;
 
 			changed = true;
+
+			/*
+			 * New multixact might have remaining XID older than
+			 * NewRelfrozenxid
+			 */
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 	}
 	else if (TransactionIdIsNormal(xid))
@@ -6522,7 +6584,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			freeze_xmax = true;
 		}
 		else
+		{
 			freeze_xmax = false;
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			{
+				/* won't be frozen, but older than current NewRelfrozenxid */
+				*NewRelfrozenxid = xid;
+			}
+		}
 	}
 	else if ((tuple->t_infomask & HEAP_XMAX_INVALID) ||
 			 !TransactionIdIsValid(HeapTupleHeaderGetRawXmax(tuple)))
@@ -6569,6 +6638,9 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 		 * was removed in PostgreSQL 9.0.  Note that if we were to respect
 		 * cutoff_xid here, we'd need to make surely to clear totally_frozen
 		 * when we skipped freezing on that basis.
+		 *
+		 * Since we always freeze here, NewRelfrozenxid doesn't need to be
+		 * maintained.
 		 */
 		if (TransactionIdIsNormal(xid))
 		{
@@ -6646,11 +6718,14 @@ heap_freeze_tuple(HeapTupleHeader tuple,
 	xl_heap_freeze_tuple frz;
 	bool		do_freeze;
 	bool		tuple_totally_frozen;
+	TransactionId NewRelfrozenxid = FirstNormalTransactionId;
+	MultiXactId NewRelminmxid = FirstMultiXactId;
 
 	do_freeze = heap_prepare_freeze_tuple(tuple,
 										  relfrozenxid, relminmxid,
 										  cutoff_xid, cutoff_multi,
-										  &frz, &tuple_totally_frozen);
+										  &frz, &tuple_totally_frozen,
+										  &NewRelfrozenxid, &NewRelminmxid);
 
 	/*
 	 * Note that because this is not a WAL-logged operation, we don't need to
@@ -7080,6 +7155,15 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  * Check to see whether any of the XID fields of a tuple (xmin, xmax, xvac)
  * are older than the specified cutoff XID or MultiXactId.  If so, return true.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will never freeze any of the XIDs from the tuple, even when we say
+ * that they should.  If caller opts to go with our recommendation to freeze,
+ * then it must account for the fact that it shouldn't trust how we've set
+ * NewRelfrozenxid/NewRelminmxid.  (In practice aggressive VACUUMs always take
+ * our recommendation because they must, and non-aggressive VACUUMs always opt
+ * to not freeze, preferring to ratchet back NewRelfrozenxid instead).
+ *
  * It doesn't matter whether the tuple is alive or dead, we are checking
  * to see if a tuple needs to be removed or frozen to avoid wraparound.
  *
@@ -7088,74 +7172,86 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  */
 bool
 heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-						MultiXactId cutoff_multi, Buffer buf)
+						MultiXactId cutoff_multi,
+						TransactionId *NewRelfrozenxid,
+						MultiXactId *NewRelminmxid, Buffer buf)
 {
 	TransactionId xid;
+	bool		needs_freeze = false;
 
 	xid = HeapTupleHeaderGetXmin(tuple);
-	if (TransactionIdIsNormal(xid) &&
-		TransactionIdPrecedes(xid, cutoff_xid))
-		return true;
+	if (TransactionIdIsNormal(xid))
+	{
+		if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			*NewRelfrozenxid = xid;
+		if (TransactionIdPrecedes(xid, cutoff_xid))
+			needs_freeze = true;
+	}
 
 	/*
 	 * The considerations for multixacts are complicated; look at
 	 * heap_prepare_freeze_tuple for justifications.  This routine had better
 	 * be in sync with that one!
+	 *
+	 * (Actually, we maintain NewRelminmxid differently here, because we
+	 * assume that XIDs that should be frozen according to cutoff_xid won't
+	 * be, whereas heap_prepare_freeze_tuple makes the opposite assumption.)
 	 */
 	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
 	{
 		MultiXactId multi;
+		MultiXactMember *members;
+		int			nmembers;
 
 		multi = HeapTupleHeaderGetRawXmax(tuple);
-		if (!MultiXactIdIsValid(multi))
-		{
-			/* no xmax set, ignore */
-			;
-		}
-		else if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
+		if (MultiXactIdIsValid(multi) &&
+			MultiXactIdPrecedes(multi, *NewRelminmxid))
+			*NewRelminmxid = multi;
+
+		if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
 			return true;
 		else if (MultiXactIdPrecedes(multi, cutoff_multi))
-			return true;
-		else
+			needs_freeze = true;
+
+		/* need to check whether any member of the mxact is too old */
+		nmembers = GetMultiXactIdMembers(multi, &members, false,
+										 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
+
+		for (int i = 0; i < nmembers; i++)
 		{
-			MultiXactMember *members;
-			int			nmembers;
-			int			i;
-
-			/* need to check whether any member of the mxact is too old */
-
-			nmembers = GetMultiXactIdMembers(multi, &members, false,
-											 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
-
-			for (i = 0; i < nmembers; i++)
-			{
-				if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-				{
-					pfree(members);
-					return true;
-				}
-			}
-			if (nmembers > 0)
-				pfree(members);
+			if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
+				needs_freeze = true;
+			if (TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
 		}
+		if (nmembers > 0)
+			pfree(members);
 	}
 	else
 	{
 		xid = HeapTupleHeaderGetRawXmax(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
 	if (tuple->t_infomask & HEAP_MOVED)
 	{
 		xid = HeapTupleHeaderGetXvac(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
-	return false;
+	return needs_freeze;
 }
 
 /*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 539214fcb..dd557fddb 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -171,8 +171,10 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
-	/* Are FreezeLimit/MultiXactCutoff still valid? */
-	bool		freeze_cutoffs_valid;
+
+	/* Track new pg_class.relfrozenxid/pg_class.relminmxid values */
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -330,6 +332,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
 
@@ -363,8 +366,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 									   params->freeze_table_age,
 									   params->multixact_freeze_min_age,
 									   params->multixact_freeze_table_age,
-									   &OldestXmin, &FreezeLimit,
-									   &MultiXactCutoff);
+									   &OldestXmin, &OldestMxact,
+									   &FreezeLimit, &MultiXactCutoff);
 
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
@@ -471,8 +474,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
-	/* Track if cutoffs became invalid (possible in !aggressive case only) */
-	vacrel->freeze_cutoffs_valid = true;
+
+	/* Initialize values used to advance relfrozenxid/relminmxid at the end */
+	vacrel->NewRelfrozenxid = OldestXmin;
+	vacrel->NewRelminmxid = OldestMxact;
 
 	/*
 	 * Call lazy_scan_heap to perform all required heap pruning, index
@@ -525,16 +530,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
 	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
 	 * provided we didn't skip any all-visible (not all-frozen) pages using
-	 * the visibility map, and assuming that we didn't fail to get a cleanup
-	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
-	 * MultiXactCutoff) established for VACUUM operation.
+	 * the visibility map.  A non-aggressive VACUUM might only be able to
+	 * advance relfrozenxid to an XID from before FreezeLimit (or a relminmxid
+	 * from before MultiXactCutoff) when it wasn't possible to freeze some
+	 * tuples due to our inability to acquire a cleanup lock, but the effect
+	 * is usually insignificant -- NewRelfrozenxid value still has a decent
+	 * chance of being much more recent that the existing relfrozenxid.
 	 *
 	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
 	 * the rel_pages used by lazy_scan_heap, which won't match when we
 	 * happened to truncate the relation afterwards.
 	 */
-	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
-		!vacrel->freeze_cutoffs_valid)
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
@@ -551,7 +558,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff,
+							vacrel->NewRelfrozenxid, vacrel->NewRelminmxid,
 							&frozenxid_updated, &minmulti_updated, false);
 	}
 
@@ -657,17 +664,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 OldestXmin, diff);
 			if (frozenxid_updated)
 			{
-				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				diff = (int32) (vacrel->NewRelfrozenxid - vacrel->relfrozenxid);
 				appendStringInfo(&buf,
 								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
-								 FreezeLimit, diff);
+								 vacrel->NewRelfrozenxid, diff);
 			}
 			if (minmulti_updated)
 			{
-				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				diff = (int32) (vacrel->NewRelminmxid - vacrel->relminmxid);
 				appendStringInfo(&buf,
 								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
-								 MultiXactCutoff, diff);
+								 vacrel->NewRelminmxid, diff);
 			}
 			if (orig_rel_pages > 0)
 			{
@@ -1581,6 +1588,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	int			nfrozen;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1589,6 +1598,8 @@ lazy_scan_prune(LVRelState *vacrel,
 retry:
 
 	/* Initialize (or reset) page-level counters */
+	NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	NewRelminmxid = vacrel->NewRelminmxid;
 	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
@@ -1798,7 +1809,9 @@ retry:
 									  vacrel->FreezeLimit,
 									  vacrel->MultiXactCutoff,
 									  &frozen[nfrozen],
-									  &tuple_totally_frozen))
+									  &tuple_totally_frozen,
+									  &NewRelfrozenxid,
+									  &NewRelminmxid))
 		{
 			/* Will execute freeze below */
 			frozen[nfrozen++].offset = offnum;
@@ -1812,13 +1825,16 @@ retry:
 			prunestate->all_frozen = false;
 	}
 
+	vacrel->offnum = InvalidOffsetNumber;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
 	 * that remains and needs to be considered for freezing now (LP_UNUSED and
 	 * LP_REDIRECT items also remain, but are of no further interest to us).
 	 */
-	vacrel->offnum = InvalidOffsetNumber;
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
 
 	/*
 	 * Consider the need to freeze any items with tuple storage from the page
@@ -1962,9 +1978,9 @@ retry:
  * We'll always return true for a non-aggressive VACUUM, even when we know
  * that this will cause them to miss out on freezing tuples from before
  * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
- * lock.  This does mean that they definitely won't be able to advance
- * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
- * relminmxid).  Caller waits for full cleanup lock when we return false.
+ * lock.  This does mean that they will have NewRelfrozenxid ratcheting back
+ * to a known-safe value (same applies to NewRelminmxid).  Caller waits for
+ * full cleanup lock when we return false.
  *
  * See lazy_scan_prune for an explanation of hastup return flag.  The
  * hasfreespace flag instructs caller on whether or not it should do generic
@@ -1988,6 +2004,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	MultiXactId NewRelminmxid = vacrel->NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -2034,7 +2052,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
 		if (heap_tuple_needs_freeze(tupleheader,
 									vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
+									vacrel->MultiXactCutoff,
+									&NewRelfrozenxid, &NewRelminmxid, buf))
 		{
 			if (vacrel->aggressive)
 			{
@@ -2044,10 +2063,11 @@ lazy_scan_noprune(LVRelState *vacrel,
 			}
 
 			/*
-			 * Current non-aggressive VACUUM operation definitely won't be
-			 * able to advance relfrozenxid or relminmxid
+			 * A non-aggressive VACUUM doesn't have to wait on a cleanup lock
+			 * to ensure that it advances relfrozenxid to a sufficiently
+			 * recent XID that happens to be present on this page.  It can
+			 * just accept an older New/final relfrozenxid instead.
 			 */
-			vacrel->freeze_cutoffs_valid = false;
 		}
 
 		num_tuples++;
@@ -2097,6 +2117,14 @@ lazy_scan_noprune(LVRelState *vacrel,
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * We have committed to not freezing the tuples on this page (always
+	 * happens with a non-aggressive VACUUM), so make sure that the target
+	 * relfrozenxid/relminmxid values reflect the XIDs/MXIDs we encountered
+	 */
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
+
 	/*
 	 * Now save details of the LP_DEAD items from the page in vacrel (though
 	 * only when VACUUM uses two-pass strategy).
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 61ced4413..fb76aac42 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -767,6 +767,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	TupleDesc	oldTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TupleDesc	newTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TransactionId OldestXmin;
+	MultiXactId oldestMxact;
 	TransactionId FreezeXid;
 	MultiXactId MultiXactCutoff;
 	bool		use_sort;
@@ -856,8 +857,8 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * Since we're going to rewrite the whole table anyway, there's no reason
 	 * not to be aggressive about this.
 	 */
-	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
+	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0, &OldestXmin, &oldestMxact,
+						  &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index f19e4a561..44cb50bc7 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -950,10 +950,28 @@ get_all_vacuum_rels(int options)
  * The output parameters are:
  * - oldestXmin is the Xid below which tuples deleted by any xact (that
  *   committed) should be considered DEAD, not just RECENTLY_DEAD.
- * - freezeLimit is the Xid below which all Xids are replaced by
- *	 FrozenTransactionId during vacuum.
+ * - oldestMxact is the Mxid below which MultiXacts are definitely not
+ *   seen as visible by any running transaction.
+ * - freezeLimit is the Xid below which all Xids are definitely replaced by
+ *   FrozenTransactionId during aggressive vacuums.
  * - multiXactCutoff is the value below which all MultiXactIds are removed
  *   from Xmax.
+ *
+ * oldestXmin and oldestMxact can be thought of as the most recent values that
+ * can ever be passed to vac_update_relstats() as frozenxid and minmulti
+ * arguments.  These exact values will be used when no newer XIDs or
+ * MultiXacts remain in the heap relation (e.g., with an empty table).  It's
+ * typical for vacuumlazy.c caller to notice that older XIDs/Multixacts remain
+ * in the table, which will force it to use older value.  These older final
+ * values may not be any newer than the preexisting frozenxid/minmulti values
+ * from pg_class in extreme cases.  The final values are frequently fairly
+ * close to the optimal values that we give to vacuumlazy.c, though.
+ *
+ * An aggressive VACUUM always provides vac_update_relstats() arguments that
+ * are >= freezeLimit and >= multiXactCutoff.  A non-aggressive VACUUM may
+ * provide arguments that are either newer or older than freezeLimit and
+ * multiXactCutoff, or non-valid values (indicating that pg_class level
+ * cutoffs cannot be advanced at all).
  */
 bool
 vacuum_set_xid_limits(Relation rel,
@@ -962,6 +980,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_min_age,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
+					  MultiXactId *oldestMxact,
 					  TransactionId *freezeLimit,
 					  MultiXactId *multiXactCutoff)
 {
@@ -970,7 +989,6 @@ vacuum_set_xid_limits(Relation rel,
 	int			effective_multixact_freeze_max_age;
 	TransactionId limit;
 	TransactionId safeLimit;
-	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
 	int			freezetable;
@@ -1066,9 +1084,11 @@ vacuum_set_xid_limits(Relation rel,
 						 effective_multixact_freeze_max_age / 2);
 	Assert(mxid_freezemin >= 0);
 
+	/* Remember for caller */
+	*oldestMxact = GetOldestMultiXactId();
+
 	/* compute the cutoff multi, being careful to generate a valid value */
-	oldestMxact = GetOldestMultiXactId();
-	mxactLimit = oldestMxact - mxid_freezemin;
+	mxactLimit = *oldestMxact - mxid_freezemin;
 	if (mxactLimit < FirstMultiXactId)
 		mxactLimit = FirstMultiXactId;
 
@@ -1083,8 +1103,8 @@ vacuum_set_xid_limits(Relation rel,
 				(errmsg("oldest multixact is far in the past"),
 				 errhint("Close open transactions with multixacts soon to avoid wraparound problems.")));
 		/* Use the safe limit, unless an older mxact is still running */
-		if (MultiXactIdPrecedes(oldestMxact, safeMxactLimit))
-			mxactLimit = oldestMxact;
+		if (MultiXactIdPrecedes(*oldestMxact, safeMxactLimit))
+			mxactLimit = *oldestMxact;
 		else
 			mxactLimit = safeMxactLimit;
 	}
-- 
2.30.2

From 0ebde19d306488c59d8b3b6e0913c5bb51c5c5e6 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 13 Dec 2021 15:00:49 -0800
Subject: [PATCH v6 5/5] Freeze tuples early to advance relfrozenxid.

Freeze whenever pruning modified the page, or whenever we see that we're
going to mark the page all-visible without also marking it all-frozen.

There has been plenty of discussion of early/opportunistic freezing in
the past.  It is generally considered important as a way of minimizing
repeated dirtying of heap pages (or the total volume of FPIs in the WAL
stream) over time.  While that goal is certainly very important, this
patch has another priority: making VACUUM advance relfrozenxid sooner
and more frequently.

The overall effect is that tables like pgbench's history table can be
vacuumed very frequently, and have most individual vacuum operations
generate 0 FPIs in WAL -- they will never need an aggressive VACUUM.

GUCs like vacuum_freeze_min_age never made much sense after the freeze
map work in PostgreSQL 9.6.  The default is 50 million transactions,
which current tends to result in our being unable to freeze tuples
before the page is marked all-visible (but not all-frozen).  This
creates a huge performance cliff later on, during the first aggressive
VACUUM.  Freezing early effectively avoids accumulating "debt" from very
old unfrozen tuples.
---
 src/include/access/heapam.h          |  1 +
 src/backend/access/heap/pruneheap.c  |  8 ++-
 src/backend/access/heap/vacuumlazy.c | 87 +++++++++++++++++++++++++---
 3 files changed, 88 insertions(+), 8 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index d35402f9f..ba094507c 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -188,6 +188,7 @@ extern int	heap_page_prune(Relation relation, Buffer buffer,
 							struct GlobalVisState *vistest,
 							TransactionId old_snap_xmin,
 							TimestampTz old_snap_ts_ts,
+							bool *modified,
 							int	*nnewlpdead,
 							OffsetNumber *off_loc);
 extern void heap_page_prune_execute(Buffer buffer,
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 3201fcc52..7d2b72e89 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -202,11 +202,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
 		 */
 		if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
 		{
+			bool	modified;
 			int		ndeleted,
 					nnewlpdead;
 
 			ndeleted = heap_page_prune(relation, buffer, vistest, limited_xmin,
-									   limited_ts, &nnewlpdead, NULL);
+									   limited_ts, &modified, &nnewlpdead, NULL);
 
 			/*
 			 * Report the number of tuples reclaimed to pgstats.  This is
@@ -264,6 +265,7 @@ heap_page_prune(Relation relation, Buffer buffer,
 				GlobalVisState *vistest,
 				TransactionId old_snap_xmin,
 				TimestampTz old_snap_ts,
+				bool *modified,
 				int	*nnewlpdead,
 				OffsetNumber *off_loc)
 {
@@ -445,6 +447,8 @@ heap_page_prune(Relation relation, Buffer buffer,
 
 			PageSetLSN(BufferGetPage(buffer), recptr);
 		}
+
+		*modified = true;
 	}
 	else
 	{
@@ -457,12 +461,14 @@ heap_page_prune(Relation relation, Buffer buffer,
 		 * point in repeating the prune/defrag process until something else
 		 * happens to the page.
 		 */
+		*modified = false;
 		if (((PageHeader) page)->pd_prune_xid != prstate.new_prune_xid ||
 			PageIsFull(page))
 		{
 			((PageHeader) page)->pd_prune_xid = prstate.new_prune_xid;
 			PageClearFull(page);
 			MarkBufferDirtyHint(buffer, true);
+			*modified = true;
 		}
 	}
 
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index dd557fddb..a7704977a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -168,6 +168,7 @@ typedef struct LVRelState
 
 	/* VACUUM operation's cutoff for pruning */
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
@@ -355,11 +356,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/*
 	 * Get cutoffs that determine which tuples we need to freeze during the
-	 * VACUUM operation.
+	 * VACUUM operation.  This includes information that is used during
+	 * opportunistic freezing, where the most aggressive possible cutoffs
+	 * (OldestXmin and OldestMxact) are used for some heap pages, based on
+	 * considerations about cost.
 	 *
 	 * Also determines if this is to be an aggressive VACUUM.  This will
 	 * eventually be required for any table where (for whatever reason) no
 	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
+	 * This used to be much more common, but we now work hard to advance
+	 * relfrozenxid in non-aggressive VACUUMs.
 	 */
 	aggressive = vacuum_set_xid_limits(rel,
 									   params->freeze_min_age,
@@ -472,6 +478,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/* Set cutoffs for entire VACUUM */
 	vacrel->OldestXmin = OldestXmin;
+	vacrel->OldestMxact = OldestMxact;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
 
@@ -1590,6 +1597,10 @@ lazy_scan_prune(LVRelState *vacrel,
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 	TransactionId NewRelfrozenxid;
 	MultiXactId NewRelminmxid;
+	bool		modified;
+	TransactionId FreezeLimit = vacrel->FreezeLimit;
+	MultiXactId MultiXactCutoff = vacrel->MultiXactCutoff;
+	bool		earlyfreezing = false;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1616,8 +1627,19 @@ retry:
 	 * that were deleted from indexes.
 	 */
 	tuples_deleted = heap_page_prune(rel, buf, vistest,
-									 InvalidTransactionId, 0, &nnewlpdead,
-									 &vacrel->offnum);
+									 InvalidTransactionId, 0, &modified,
+									 &nnewlpdead, &vacrel->offnum);
+
+	/*
+	 * If page was modified during pruning, then perform early freezing
+	 * opportunistically
+	 */
+	if (!earlyfreezing && modified)
+	{
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+	}
 
 	/*
 	 * Now scan the page to collect LP_DEAD items and check for tuples
@@ -1672,7 +1694,7 @@ retry:
 		if (ItemIdIsDead(itemid))
 		{
 			deadoffsets[lpdead_items++] = offnum;
-			prunestate->all_visible = false;
+			/* Don't set all_visible to false just yet */
 			prunestate->has_lpdead_items = true;
 			continue;
 		}
@@ -1806,8 +1828,8 @@ retry:
 		if (heap_prepare_freeze_tuple(tuple.t_data,
 									  vacrel->relfrozenxid,
 									  vacrel->relminmxid,
-									  vacrel->FreezeLimit,
-									  vacrel->MultiXactCutoff,
+									  FreezeLimit,
+									  MultiXactCutoff,
 									  &frozen[nfrozen],
 									  &tuple_totally_frozen,
 									  &NewRelfrozenxid,
@@ -1827,6 +1849,57 @@ retry:
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * Reconsider applying early freezing before committing to processing the
+	 * page as currently planned.  There are 2 reasons to change our mind:
+	 *
+	 * 1. The standard FreezeLimit cutoff generally indicates that we should
+	 * freeze XIDs that are more than freeze_min_age XIDs in the past
+	 * (relative to OldestXmin).  But that should only be treated as a rough
+	 * guideline; it makes sense to freeze all eligible tuples on pages where
+	 * we're going to freeze at least one in any case.
+	 *
+	 * 2. If the page is now eligible to be marked all_visible, but is not
+	 * also eligible to be marked all_frozen, then we freeze early to make
+	 * sure that the page becomes all_frozen.  We should avoid building up
+	 * "freeze debt" that can only be paid off by an aggressive VACUUM, later
+	 * on.  This makes it much less likely that an aggressive VACUUM will ever
+	 * be required.
+	 *
+	 * Note: We deliberately track all_visible in a way that excludes LP_DEAD
+	 * items here.  Any page that is "all_visible for tuples with storage"
+	 * will be eligible to have its visibility map bit set during the ongoing
+	 * VACUUM, one way or another.  LP_DEAD items only make it unsafe to set
+	 * the page all_visible during the first heap pass, but the second heap
+	 * pass should be able to perform equivalent processing. (The second heap
+	 * pass cannot freeze tuples, though.)
+	 */
+	if (!earlyfreezing &&
+		((nfrozen > 0 && nfrozen < num_tuples) ||
+		 (prunestate->all_visible && !prunestate->all_frozen)))
+	{
+		/*
+		 * XXX Need to worry about leaking MultiXacts in FreezeMultiXactId()
+		 * now (via heap_prepare_freeze_tuple calls)?  That was already
+		 * possible, but presumably this makes it much more likely.
+		 *
+		 * On the other hand, that's only possible when we need to replace an
+		 * existing MultiXact with a new one.  Even then, we won't have
+		 * preallocated a new MultiXact (which we now risk leaking) if there
+		 * was only one remaining XID, and the XID is for an updater (we'll
+		 * only prepare to replace xmax with the XID directly).  So maybe it's
+		 * still a narrow enough problem to be ignored.
+		 */
+		earlyfreezing = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+		goto retry;
+	}
+
+	/* Time to define all_visible in a way that accounts for LP_DEAD items */
+	if (lpdead_items > 0)
+		prunestate->all_visible = false;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
@@ -1872,7 +1945,7 @@ retry:
 		{
 			XLogRecPtr	recptr;
 
-			recptr = log_heap_freeze(vacrel->rel, buf, vacrel->FreezeLimit,
+			recptr = log_heap_freeze(vacrel->rel, buf, FreezeLimit,
 									 frozen, nfrozen);
 			PageSetLSN(page, recptr);
 		}
-- 
2.30.2

From b1c5bda8102a3a8e7c7b5548b8110485092d3795 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 11 Dec 2021 17:39:45 -0800
Subject: [PATCH v6 3/5] Consolidate VACUUM xid cutoff logic.

Push the logic for determining whether or not any given VACUUM operation
will be aggressive down into vacuum_set_xid_limits().  This makes its
function signature significantly simpler.

This refactoring work will make it easier to set/return an "oldestMxact"
value the function's vacuumlazy.c caller in a later commit that teaches
VACUUM to intelligently set relfrozenxid and relminmxid to the oldest
real remain xid/MultiXactId.

A VACUUM operation's oldestMxact can be thought of as the MultiXactId
equivalent of its OldestXmin: just as OldestXmin is used as our initial
target relfrozenxid (which we'll ratchet back as the VACUUM progresses
and notices that it'll leave older XIDs in place), oldestMxact will be
our initial target MultiXactId (for a target MultiXactId that is itself
ratcheted back in the same way).
---
 src/include/commands/vacuum.h        |   6 +-
 src/backend/access/heap/vacuumlazy.c |  32 +++----
 src/backend/commands/cluster.c       |   3 +-
 src/backend/commands/vacuum.c        | 134 +++++++++++++--------------
 4 files changed, 79 insertions(+), 96 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index e5e548d6b..d64f6268f 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -286,15 +286,13 @@ extern void vac_update_relstats(Relation relation,
 								bool *frozenxid_updated,
 								bool *minmulti_updated,
 								bool in_outer_xact);
-extern void vacuum_set_xid_limits(Relation rel,
+extern bool vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
 								  TransactionId *freezeLimit,
-								  TransactionId *xidFullScanLimit,
-								  MultiXactId *multiXactCutoff,
-								  MultiXactId *mxactFullScanLimit);
+								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
 									  MultiXactId relminmxid);
 extern void vac_update_datfrozenxid(void);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 49847bc00..539214fcb 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -323,8 +323,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
-	TransactionId xidFullScanLimit;
-	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
@@ -352,24 +350,22 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
 								  RelationGetRelid(rel));
 
-	vacuum_set_xid_limits(rel,
-						  params->freeze_min_age,
-						  params->freeze_table_age,
-						  params->multixact_freeze_min_age,
-						  params->multixact_freeze_table_age,
-						  &OldestXmin, &FreezeLimit, &xidFullScanLimit,
-						  &MultiXactCutoff, &mxactFullScanLimit);
-
 	/*
-	 * We request an aggressive scan if the table's frozen Xid is now older
-	 * than or equal to the requested Xid full-table scan limit; or if the
-	 * table's minimum MultiXactId is older than or equal to the requested
-	 * mxid full-table scan limit; or if DISABLE_PAGE_SKIPPING was specified.
+	 * Get cutoffs that determine which tuples we need to freeze during the
+	 * VACUUM operation.
+	 *
+	 * Also determines if this is to be an aggressive VACUUM.  This will
+	 * eventually be required for any table where (for whatever reason) no
+	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
 	 */
-	aggressive = TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
-											   xidFullScanLimit);
-	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
-											  mxactFullScanLimit);
+	aggressive = vacuum_set_xid_limits(rel,
+									   params->freeze_min_age,
+									   params->freeze_table_age,
+									   params->multixact_freeze_min_age,
+									   params->multixact_freeze_table_age,
+									   &OldestXmin, &FreezeLimit,
+									   &MultiXactCutoff);
+
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
 	{
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 61853e6de..61ced4413 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -857,8 +857,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * not to be aggressive about this.
 	 */
 	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, NULL, &MultiXactCutoff,
-						  NULL);
+						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index b72ce01c5..f19e4a561 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -942,25 +942,20 @@ get_all_vacuum_rels(int options)
  *
  * Input parameters are the target relation, applicable freeze age settings.
  *
+ * Return value indicates whether caller should do an aggressive VACUUM or
+ * not.  This is a VACUUM that cannot skip any pages using the visibility map
+ * (except all-frozen pages), which is guaranteed to be able to advance
+ * relfrozenxid and relminmxid.
+ *
  * The output parameters are:
- * - oldestXmin is the cutoff value used to distinguish whether tuples are
- *	 DEAD or RECENTLY_DEAD (see HeapTupleSatisfiesVacuum).
+ * - oldestXmin is the Xid below which tuples deleted by any xact (that
+ *   committed) should be considered DEAD, not just RECENTLY_DEAD.
  * - freezeLimit is the Xid below which all Xids are replaced by
  *	 FrozenTransactionId during vacuum.
- * - xidFullScanLimit (computed from freeze_table_age parameter)
- *	 represents a minimum Xid value; a table whose relfrozenxid is older than
- *	 this will have a full-table vacuum applied to it, to freeze tuples across
- *	 the whole table.  Vacuuming a table younger than this value can use a
- *	 partial scan.
- * - multiXactCutoff is the value below which all MultiXactIds are removed from
- *	 Xmax.
- * - mxactFullScanLimit is a value against which a table's relminmxid value is
- *	 compared to produce a full-table vacuum, as with xidFullScanLimit.
- *
- * xidFullScanLimit and mxactFullScanLimit can be passed as NULL if caller is
- * not interested.
+ * - multiXactCutoff is the value below which all MultiXactIds are removed
+ *   from Xmax.
  */
-void
+bool
 vacuum_set_xid_limits(Relation rel,
 					  int freeze_min_age,
 					  int freeze_table_age,
@@ -968,9 +963,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
 					  TransactionId *freezeLimit,
-					  TransactionId *xidFullScanLimit,
-					  MultiXactId *multiXactCutoff,
-					  MultiXactId *mxactFullScanLimit)
+					  MultiXactId *multiXactCutoff)
 {
 	int			freezemin;
 	int			mxid_freezemin;
@@ -980,6 +973,7 @@ vacuum_set_xid_limits(Relation rel,
 	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
+	int			freezetable;
 
 	/*
 	 * We can always ignore processes running lazy vacuum.  This is because we
@@ -1097,64 +1091,60 @@ vacuum_set_xid_limits(Relation rel,
 
 	*multiXactCutoff = mxactLimit;
 
-	if (xidFullScanLimit != NULL)
-	{
-		int			freezetable;
+	/*
+	 * Done setting output parameters; just need to figure out if caller needs
+	 * to do an aggressive VACUUM or not.
+	 *
+	 * Determine the table freeze age to use: as specified by the caller, or
+	 * vacuum_freeze_table_age, but in any case not more than
+	 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
+	 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
+	 * before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_freeze_table_age;
+	freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		Assert(mxactFullScanLimit != NULL);
+	/*
+	 * Compute XID limit causing an aggressive vacuum, being careful not to
+	 * generate a "permanent" XID
+	 */
+	limit = ReadNextTransactionId() - freezetable;
+	if (!TransactionIdIsNormal(limit))
+		limit = FirstNormalTransactionId;
+	if (TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
+									  limit))
+		return true;
 
-		/*
-		 * Determine the table freeze age to use: as specified by the caller,
-		 * or vacuum_freeze_table_age, but in any case not more than
-		 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
-		 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
-		 * before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_freeze_table_age;
-		freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
+	/*
+	 * Similar to the above, determine the table freeze age to use for
+	 * multixacts: as specified by the caller, or
+	 * vacuum_multixact_freeze_table_age, but in any case not more than
+	 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have e.g.
+	 * nightly VACUUM schedule, the nightly VACUUM gets a chance to freeze
+	 * multixacts before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = multixact_freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_multixact_freeze_table_age;
+	freezetable = Min(freezetable,
+					  effective_multixact_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		/*
-		 * Compute XID limit causing a full-table vacuum, being careful not to
-		 * generate a "permanent" XID.
-		 */
-		limit = ReadNextTransactionId() - freezetable;
-		if (!TransactionIdIsNormal(limit))
-			limit = FirstNormalTransactionId;
+	/*
+	 * Compute MultiXact limit causing an aggressive vacuum, being careful to
+	 * generate a valid MultiXact value
+	 */
+	mxactLimit = ReadNextMultiXactId() - freezetable;
+	if (mxactLimit < FirstMultiXactId)
+		mxactLimit = FirstMultiXactId;
+	if (MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
+									mxactLimit))
+		return true;
 
-		*xidFullScanLimit = limit;
-
-		/*
-		 * Similar to the above, determine the table freeze age to use for
-		 * multixacts: as specified by the caller, or
-		 * vacuum_multixact_freeze_table_age, but in any case not more than
-		 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have
-		 * e.g. nightly VACUUM schedule, the nightly VACUUM gets a chance to
-		 * freeze multixacts before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = multixact_freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_multixact_freeze_table_age;
-		freezetable = Min(freezetable,
-						  effective_multixact_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
-
-		/*
-		 * Compute MultiXact limit causing a full-table vacuum, being careful
-		 * to generate a valid MultiXact value.
-		 */
-		mxactLimit = ReadNextMultiXactId() - freezetable;
-		if (mxactLimit < FirstMultiXactId)
-			mxactLimit = FirstMultiXactId;
-
-		*mxactFullScanLimit = mxactLimit;
-	}
-	else
-	{
-		Assert(mxactFullScanLimit == NULL);
-	}
+	return false;
 }
 
 /*
-- 
2.30.2

From ba4d5dd0b280d189d4c1dbe45a7dbb1970fb66c6 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v6 1/5] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

We now also collect LP_DEAD items in the dead_items array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

We no longer report on "pin skipped pages" in log output.  A later patch
will add back an improved version of the same instrumentation.  We don't
want to show any information about any failures to acquire cleanup locks
unless we actually failed to do useful work as a consequence.  A page
that we could not acquire a cleanup lock on is now treated as equivalent
to any other scanned page in most cases.
---
 src/backend/access/heap/vacuumlazy.c          | 805 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 517 insertions(+), 297 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1749cc2a4..16f88bab0 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -143,6 +143,10 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
+	/* Use visibility map to skip? (disabled via reloption) */
+	bool		skipwithvm;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -167,6 +171,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -188,10 +194,8 @@ typedef struct LVRelState
 	 */
 	VacDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -204,6 +208,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -240,19 +245,22 @@ typedef struct LVSavedErrInfo
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup, bool *hasfreespace);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void lazy_cleanup_all_indexes(LVRelState *vacrel);
 static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -307,16 +315,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive,
+				skipwithvm;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -359,8 +366,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 											   xidFullScanLimit);
 	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
 											  mxactFullScanLimit);
+	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
+	{
+		/*
+		 * Force aggressive mode, and disable skipping blocks using the
+		 * visibility map (even those set all-frozen)
+		 */
 		aggressive = true;
+		skipwithvm = false;
+	}
 
 	/*
 	 * Setup error traceback support for ereport() first.  The idea is to set
@@ -423,6 +438,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	Assert(params->index_cleanup != VACOPTVALUE_UNSPECIFIED);
 	Assert(params->truncate != VACOPTVALUE_UNSPECIFIED &&
 		   params->truncate != VACOPTVALUE_AUTO);
+	vacrel->aggressive = aggressive;
+	vacrel->skipwithvm = skipwithvm;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 	vacrel->do_index_vacuuming = true;
@@ -454,35 +471,23 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	/*
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, params->nworkers);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
@@ -508,28 +513,44 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
+	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
+	 * provided we didn't skip any all-visible (not all-frozen) pages using
+	 * the visibility map, and assuming that we didn't fail to get a cleanup
+	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
+	 * MultiXactCutoff) established for VACUUM operation.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_heap, which won't match when we
+	 * happened to truncate the relation afterwards.
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -557,7 +578,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -609,10 +629,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -620,7 +639,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->pages_removed;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -737,7 +755,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, int nworkers)
 {
 	VacDeadItems *dead_items;
 	BlockNumber nblocks,
@@ -756,14 +774,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	GlobalVisState *vistest;
 
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
-	next_unskippable_block = 0;
-	next_failsafe_block = 0;
-	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -787,14 +800,16 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * dangerously old.
 	 */
 	lazy_check_wraparound_failsafe(vacrel);
+	next_failsafe_block = 0;
 
 	/*
 	 * Allocate the space for dead_items.  Note that this handles parallel
 	 * VACUUM initialization as part of allocating shared memory space used
 	 * for dead_items.
 	 */
-	dead_items_alloc(vacrel, params->nworkers);
+	dead_items_alloc(vacrel, nworkers);
 	dead_items = vacrel->dead_items;
+	next_fsm_block_to_vacuum = 0;
 
 	/* Report that we're scanning the heap, advertising total # of blocks */
 	initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
@@ -803,7 +818,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/*
-	 * Except when aggressive is set, we want to skip pages that are
+	 * Set things up for skipping blocks using visibility map.
+	 *
+	 * Except when vacrel->aggressive is set, we want to skip pages that are
 	 * all-visible according to the visibility map, but only when we can skip
 	 * at least SKIP_PAGES_THRESHOLD consecutive pages.  Since we're reading
 	 * sequentially, the OS should be doing readahead for us, so there's no
@@ -812,8 +829,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * page means that we can't update relfrozenxid, so we only want to do it
 	 * if we can skip a goodly number of pages.
 	 *
-	 * When aggressive is set, we can't skip pages just because they are
-	 * all-visible, but we can still skip pages that are all-frozen, since
+	 * When vacrel->aggressive is set, we can't skip pages just because they
+	 * are all-visible, but we can still skip pages that are all-frozen, since
 	 * such pages do not need freezing and do not affect the value that we can
 	 * safely set for relfrozenxid or relminmxid.
 	 *
@@ -836,17 +853,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
-	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+	next_unskippable_block = 0;
+	if (vacrel->skipwithvm)
 	{
 		while (next_unskippable_block < nblocks)
 		{
@@ -855,7 +864,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vmstatus = visibilitymap_get_status(vacrel->rel,
 												next_unskippable_block,
 												&vmbuffer);
-			if (aggressive)
+			if (vacrel->aggressive)
 			{
 				if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
 					break;
@@ -882,13 +891,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
@@ -898,7 +900,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		{
 			/* Time to advance next_unskippable_block */
 			next_unskippable_block++;
-			if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+			if (vacrel->skipwithvm)
 			{
 				while (next_unskippable_block < nblocks)
 				{
@@ -907,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 					vmskipflags = visibilitymap_get_status(vacrel->rel,
 														   next_unskippable_block,
 														   &vmbuffer);
-					if (aggressive)
+					if (vacrel->aggressive)
 					{
 						if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
 							break;
@@ -936,19 +938,25 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			 * it's not all-visible.  But in an aggressive vacuum we know only
 			 * that it's not all-frozen, so it might still be all-visible.
 			 */
-			if (aggressive && VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
+			if (vacrel->aggressive &&
+				VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
 				all_visible_according_to_vm = true;
 		}
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current page can be skipped if we've seen a long enough run
+			 * of skippable blocks to justify skipping it -- provided it's not
+			 * the last page in the relation (according to rel_pages/nblocks).
+			 *
+			 * We always scan the table's last page to determine whether it
+			 * has tuples or not, even if it would otherwise be skipped
+			 * (unless we're skipping every single page in the relation). This
+			 * avoids having lazy_truncate_heap() take access-exclusive lock
+			 * on the table to attempt a truncation that just fails
+			 * immediately because there are tuples on the last page.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks && blkno < nblocks - 1)
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -957,18 +965,32 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * careful to count it as a skipped all-frozen page in that
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
-				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * know whether it was initially all-frozen, so we have to
+				 * recheck.
 				 */
-				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+				if (vacrel->aggressive ||
+					VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise it must be an all-visible (and possibly even
+			 * all-frozen) page that we decided to process regardless
+			 * (SKIP_PAGES_THRESHOLD must not have been crossed).
+			 */
 			all_visible_according_to_vm = true;
 		}
 
 		vacuum_delay_point();
 
+		/*
+		 * We're not skipping this page using the visibility map, and so it is
+		 * (by definition) a scanned page.  Any tuples from this page are now
+		 * guaranteed to be counted below, after some preparatory checks.
+		 */
+		vacrel->scanned_pages++;
+
 		/*
 		 * Regularly check if wraparound failsafe should trigger.
 		 *
@@ -1023,174 +1045,78 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
-		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
-		 * already have the correct page pinned anyway.  However, it's
-		 * possible that (a) next_unskippable_block is covered by a different
-		 * VM page than the current block or (b) we released our pin and did a
-		 * cycle of index vacuuming.
+		 * already have the correct page pinned anyway.
 		 */
 		visibilitymap_pin(vacrel->rel, blkno, &vmbuffer);
 
+		/* Finished preparatory checks.  Actually scan the page. */
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing using lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
-			bool		hastup;
+			bool		hastup,
+						hasfreespace;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
-			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				continue;
-			}
-
-			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
-			 */
 			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
+				/* Processed as new/empty page (lock and pin released) */
 				continue;
 			}
-			if (!aggressive)
+
+			/* Collect LP_DEAD items in dead_items array, count tuples */
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
+								  &hasfreespace))
 			{
+				Size		freespace;
+
 				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
+				 * Processed page successfully (without cleanup lock) -- just
+				 * need to perform rel truncation and FSM steps, much like the
+				 * lazy_scan_prune case.  Don't bother trying to match its
+				 * visibility map setting steps, though.
 				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
 				if (hastup)
 					vacrel->nonempty_pages = blkno + 1;
+				if (hasfreespace)
+					freespace = PageGetHeapFreeSpace(page);
+				UnlockReleaseBuffer(buf);
+				if (hasfreespace)
+					RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
 			}
+
+			/*
+			 * lazy_scan_noprune could not do all required processing.  Wait
+			 * for a cleanup lock, and call lazy_scan_prune in the usual way.
+			 */
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Processed as new/empty page (lock and pin released) */
 			continue;
 		}
 
 		/*
-		 * Prune and freeze tuples.
+		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
 		 * dead_items array.  This includes LP_DEAD line pointers that we
@@ -1398,7 +1324,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1447,6 +1373,137 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		update_index_statistics(vacrel);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller must hold at least a shared lock.  We might need to escalate the
+ * lock in that case, so the type of lock caller holds needs to be specified
+ * using 'sharelock' argument.
+ *
+ * Returns false in common case where caller should go on to call
+ * lazy_scan_prune (or lazy_scan_noprune).  Otherwise returns true, indicating
+ * that lazy_scan_heap is done processing the page, releasing lock on caller's
+ * behalf.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never discover the space on a promoted standby.
+		 * The harm of repeated checking ought to normally not be too bad. The
+		 * space usually should be used at some point, otherwise there
+		 * wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1491,6 +1548,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
 	maxoff = PageGetMaxOffsetNumber(page);
 
 retry:
@@ -1553,10 +1612,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -1843,6 +1901,226 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() variant without pruning
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED in lazy_vacuum_heap_page.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).
+ *
+ * Returns true to indicate that all required processing has been performed.
+ * We'll always return true for a non-aggressive VACUUM, even when we know
+ * that this will cause them to miss out on freezing tuples from before
+ * vacrel->FreezeLimit cutoff -- they should never have to wait for a cleanup
+ * lock.  This does mean that they definitely won't be able to advance
+ * relfrozenxid opportunistically (same applies to vacrel->MultiXactCutoff and
+ * relminmxid).  Caller waits for full cleanup lock when we return false.
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.  The
+ * hasfreespace flag instructs caller on whether or not it should do generic
+ * FSM processing for page, which is determined based on almost the same
+ * criteria as the lazy_scan_prune case.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup,
+				  bool *hasfreespace)
+{
+	OffsetNumber offnum,
+				maxoff;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
+	*hastup = false;			/* for now */
+	*hasfreespace = false;		/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		*hastup = true;		/* page prevents rel truncation */
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (heap_tuple_needs_freeze(tupleheader,
+									vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			/*
+			 * Current non-aggressive VACUUM operation definitely won't be
+			 * able to advance relfrozenxid or relminmxid
+			 */
+			vacrel->freeze_cutoffs_valid = false;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+
+				/*
+				 * There is some useful work for pruning to do, that won't be
+				 * done due to failure to get a cleanup lock.
+				 *
+				 * TODO Add dedicated instrumentation for this case
+				 */
+				break;
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in vacrel (though
+	 * only when VACUUM uses two-pass strategy).
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/*
+		 * Using one-pass strategy.
+		 *
+		 * We are not prepared to handle the corner case where a single pass
+		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
+		 * items.
+		 */
+		if (lpdead_items > 0)
+			*hastup = true;
+		*hasfreespace = true;
+		num_tuples += lpdead_items;
+		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+	}
+	else if (lpdead_items > 0)
+	{
+		VacDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * Caller won't be vacuuming this page later, so tell it to record
+		 * page's freespace in the FSM now
+		 */
+		*hasfreespace = true;
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Main entry point for index vacuuming and heap vacuuming.
  *
@@ -2286,67 +2564,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2412,7 +2629,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -2429,7 +2646,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 		/* Outsource everything to parallel variant */
 		parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
 											vacrel->num_index_scans,
-											(vacrel->tupcount_pages < vacrel->rel_pages));
+											(vacrel->scanned_pages < vacrel->rel_pages));
 	}
 }
 
@@ -2536,7 +2753,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From a3c4fdb7f87dcc96c59fc9bcb584c622124b6215 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 21 Nov 2021 14:47:11 -0800
Subject: [PATCH v6 2/5] Improve VACUUM instrumentation.

Add instrumentation of "missed dead tuples", and the number of pages
that had at least one such tuple.  These are fully DEAD (not just
RECENTLY_DEAD) tuples with storage that could not be pruned due to an
inability to acquire a cleanup lock.  This is a replacement for the
"skipped due to pin" instrumentation removed by the previous commit.
Note that the new instrumentation doesn't say anything about pages that
we failed to acquire a cleanup lock on when we see that there were no
missed dead tuples on the page.

Also report on visibility map pages skipped by VACUUM, without regard
for whether the pages were all-frozen or just all-visible.

Also report when and how relfrozenxid is advanced by VACUUM, including
non-aggressive VACUUM.  Apart from being useful on its own, this might
enable future work that teaches non-aggressive VACUUM to be more
concerned about advancing relfrozenxid sooner rather than later.

Also report number of tuples frozen.  This will become more important
when a later patch that freezes tuples early is committed.

Also enhance how we report OldestXmin cutoff by putting it in context:
show how far behind it is at the _end_ of the VACUUM operation.
---
 src/include/commands/vacuum.h        |   2 +
 src/backend/access/heap/vacuumlazy.c | 105 +++++++++++++++++++--------
 src/backend/commands/analyze.c       |   3 +
 src/backend/commands/vacuum.c        |   9 +++
 4 files changed, 89 insertions(+), 30 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d0bdfa42..e5e548d6b 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -283,6 +283,8 @@ extern void vac_update_relstats(Relation relation,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
+								bool *frozenxid_updated,
+								bool *minmulti_updated,
 								bool in_outer_xact);
 extern void vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 16f88bab0..49847bc00 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -198,6 +198,7 @@ typedef struct LVRelState
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber pages_removed;	/* pages remove by truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
+	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
 
 	/* Statistics output by us, for table */
@@ -210,9 +211,10 @@ typedef struct LVRelState
 	int			num_index_scans;
 	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
+	int64		tuples_frozen;	/* # frozen by us */
 	int64		lpdead_items;	/* # deleted from indexes */
-	int64		new_dead_tuples;	/* new estimated total # of dead items in
-									 * table */
+	int64		recently_dead_tuples;	/* # dead, but not yet removable */
+	int64		missed_dead_tuples; /* # removable, but not removed */
 	int64		num_tuples;		/* total number of nonremovable tuples */
 	int64		live_tuples;	/* live tuples (reltuples estimate) */
 } LVRelState;
@@ -317,6 +319,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				write_rate;
 	bool		aggressive,
 				skipwithvm;
+	bool		frozenxid_updated,
+				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
@@ -538,9 +542,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
+		frozenxid_updated = minmulti_updated = false;
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							InvalidTransactionId, InvalidMultiXactId, false);
+							InvalidTransactionId, InvalidMultiXactId,
+							NULL, NULL, false);
 	}
 	else
 	{
@@ -549,7 +555,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff, false);
+							FreezeLimit, MultiXactCutoff,
+							&frozenxid_updated, &minmulti_updated, false);
 	}
 
 	/*
@@ -565,7 +572,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_report_vacuum(RelationGetRelid(rel),
 						 rel->rd_rel->relisshared,
 						 Max(new_live_tuples, 0),
-						 vacrel->new_dead_tuples);
+						 vacrel->recently_dead_tuples +
+						 vacrel->missed_dead_tuples);
 	pgstat_progress_end_command();
 
 	if (instrument)
@@ -578,6 +586,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
+			int32		diff;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -629,16 +638,41 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped using visibility map (%.2f%% of total)\n"),
 							 vacrel->pages_removed,
 							 vacrel->rel_pages,
-							 vacrel->frozenskipped_pages);
+							 orig_rel_pages - vacrel->scanned_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * (orig_rel_pages - vacrel->scanned_pages) / orig_rel_pages);
 			appendStringInfo(&buf,
-							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
+							 _("tuples: %lld removed, %lld remain (%lld newly frozen), %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
 							 (long long) vacrel->new_rel_tuples,
-							 (long long) vacrel->new_dead_tuples,
-							 OldestXmin);
+							 (long long) vacrel->tuples_frozen,
+							 (long long) vacrel->recently_dead_tuples);
+			if (vacrel->missed_dead_tuples > 0)
+				appendStringInfo(&buf,
+								 _("tuples missed: %lld dead from %u contended pages\n"),
+								 (long long) vacrel->missed_dead_tuples,
+								 vacrel->missed_dead_pages);
+			diff = (int32) (ReadNextTransactionId() - OldestXmin);
+			appendStringInfo(&buf,
+							 _("removable cutoff: %u, which is %d xids behind next\n"),
+							 OldestXmin, diff);
+			if (frozenxid_updated)
+			{
+				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				appendStringInfo(&buf,
+								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
+								 FreezeLimit, diff);
+			}
+			if (minmulti_updated)
+			{
+				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				appendStringInfo(&buf,
+								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
+								 MultiXactCutoff, diff);
+			}
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -779,13 +813,16 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	vacrel->frozenskipped_pages = 0;
 	vacrel->pages_removed = 0;
 	vacrel->lpdead_item_pages = 0;
+	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
 
 	/* Initialize instrumentation counters */
 	vacrel->num_index_scans = 0;
 	vacrel->tuples_deleted = 0;
+	vacrel->tuples_frozen = 0;
 	vacrel->lpdead_items = 0;
-	vacrel->new_dead_tuples = 0;
+	vacrel->recently_dead_tuples = 0;
+	vacrel->missed_dead_tuples = 0;
 	vacrel->num_tuples = 0;
 	vacrel->live_tuples = 0;
 
@@ -1332,7 +1369,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	 * (unlikely) scenario that new_live_tuples is -1, take it as zero.
 	 */
 	vacrel->new_rel_tuples =
-		Max(vacrel->new_live_tuples, 0) + vacrel->new_dead_tuples;
+		Max(vacrel->new_live_tuples, 0) + vacrel->recently_dead_tuples +
+		vacrel->missed_dead_tuples;
 
 	/*
 	 * Release any remaining pin on visibility map page.
@@ -1540,7 +1578,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				lpdead_items,
-				new_dead_tuples,
+				recently_dead_tuples,
 				num_tuples,
 				live_tuples;
 	int			nnewlpdead;
@@ -1557,7 +1595,7 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	tuples_deleted = 0;
 	lpdead_items = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
 	num_tuples = 0;
 	live_tuples = 0;
 
@@ -1716,11 +1754,11 @@ retry:
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * If tuple is recently deleted then we must not remove it
-				 * from relation.  (We only remove items that are LP_DEAD from
+				 * If tuple is recently dead then we must not remove it from
+				 * the relation.  (We only remove items that are LP_DEAD from
 				 * pruning.)
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				prunestate->all_visible = false;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -1895,8 +1933,9 @@ retry:
 
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
+	vacrel->tuples_frozen += nfrozen;
 	vacrel->lpdead_items += lpdead_items;
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
 }
@@ -1949,7 +1988,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	int			lpdead_items,
 				num_tuples,
 				live_tuples,
-				new_dead_tuples;
+				recently_dead_tuples,
+				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 
@@ -1961,7 +2001,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	lpdead_items = 0;
 	num_tuples = 0;
 	live_tuples = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
+	missed_dead_tuples = 0;
 
 	maxoff = PageGetMaxOffsetNumber(page);
 	for (offnum = FirstOffsetNumber;
@@ -2035,16 +2076,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 				/*
 				 * There is some useful work for pruning to do, that won't be
 				 * done due to failure to get a cleanup lock.
-				 *
-				 * TODO Add dedicated instrumentation for this case
 				 */
+				missed_dead_tuples++;
 				break;
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 * Count in recently_dead_tuples, just like lazy_scan_prune
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
 
@@ -2072,13 +2112,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 		 *
 		 * We are not prepared to handle the corner case where a single pass
 		 * strategy VACUUM cannot get a cleanup lock, and we then find LP_DEAD
-		 * items.
+		 * items.  Count the LP_DEAD items as missed_dead_tuples instead. This
+		 * is slightly dishonest, but it's better than maintaining code to do
+		 * heap vacuuming for this one narrow corner case.
 		 */
 		if (lpdead_items > 0)
 			*hastup = true;
 		*hasfreespace = true;
 		num_tuples += lpdead_items;
-		/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+		missed_dead_tuples += lpdead_items;
 	}
 	else if (lpdead_items > 0)
 	{
@@ -2113,9 +2155,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 	/*
 	 * Finally, add relevant page-local counts to whole-VACUUM counts
 	 */
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
+	vacrel->missed_dead_tuples += missed_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
+	if (missed_dead_tuples > 0)
+		vacrel->missed_dead_pages++;
 
 	/* Caller won't need to call lazy_scan_prune with same page */
 	return true;
@@ -2194,8 +2239,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * dead_items space is not CPU cache resident.
 		 *
 		 * We don't take any special steps to remember the LP_DEAD items (such
-		 * as counting them in new_dead_tuples report to the stats collector)
-		 * when the optimization is applied.  Though the accounting used in
+		 * as counting them in our final report to the stats collector) when
+		 * the optimization is applied.  Though the accounting used in
 		 * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD
 		 * items as dead rows in its own stats collector report, that's okay.
 		 * The discrepancy should be negligible.  If this optimization is ever
@@ -3322,7 +3367,7 @@ update_index_statistics(LVRelState *vacrel)
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
-							false);
+							NULL, NULL, false);
 	}
 }
 
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cc9705d06..7bf7f6e86 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -651,6 +651,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 
 		/* Same for indexes */
@@ -667,6 +668,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
+								NULL, NULL,
 								in_outer_xact);
 		}
 	}
@@ -679,6 +681,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		vac_update_relstats(onerel, -1, totalrows,
 							0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 	}
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 283ffaea7..b72ce01c5 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1315,6 +1315,7 @@ vac_update_relstats(Relation relation,
 					BlockNumber num_all_visible_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
+					bool *frozenxid_updated, bool *minmulti_updated,
 					bool in_outer_xact)
 {
 	Oid			relid = RelationGetRelid(relation);
@@ -1390,22 +1391,30 @@ vac_update_relstats(Relation relation,
 	 * This should match vac_update_datfrozenxid() concerning what we consider
 	 * to be "in the future".
 	 */
+	if (frozenxid_updated)
+		*frozenxid_updated = false;
 	if (TransactionIdIsNormal(frozenxid) &&
 		pgcform->relfrozenxid != frozenxid &&
 		(TransactionIdPrecedes(pgcform->relfrozenxid, frozenxid) ||
 		 TransactionIdPrecedes(ReadNextTransactionId(),
 							   pgcform->relfrozenxid)))
 	{
+		if (frozenxid_updated)
+			*frozenxid_updated = true;
 		pgcform->relfrozenxid = frozenxid;
 		dirty = true;
 	}
 
 	/* Similarly for relminmxid */
+	if (minmulti_updated)
+		*minmulti_updated = false;
 	if (MultiXactIdIsValid(minmulti) &&
 		pgcform->relminmxid != minmulti &&
 		(MultiXactIdPrecedes(pgcform->relminmxid, minmulti) ||
 		 MultiXactIdPrecedes(ReadNextMultiXactId(), pgcform->relminmxid)))
 	{
+		if (minmulti_updated)
+			*minmulti_updated = true;
 		pgcform->relminmxid = minmulti;
 		dirty = true;
 	}
-- 
2.30.2

From 8a6624b51960019dd3001050e06dc1716b67816f Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 22 Nov 2021 10:02:30 -0800
Subject: [PATCH v7 4/6] Loosen coupling between relfrozenxid and tuple
 freezing.

The pg_class.relfrozenxid invariant for heap relations is as follows:
relfrozenxid must be less than or equal to the oldest extant XID in the
table, and must never wraparound (it must be advanced by VACUUM before
wraparound, or in extreme cases the system must be forced to stop
allocating new XIDs).

Before now, VACUUM always set relfrozenxid to whatever value it happened
to use when determining which tuples to freeze (the VACUUM operation's
FreezeLimit cutoff).  But there was no inherent reason why the oldest
extant XID in the table should be anywhere near as old as that.
Furthermore, even if it really was almost as old as FreezeLimit, that
tells us much more about the mechanism that VACUUM used to determine
which tuples to freeze than anything else.  Depending on the details of
the table and workload, it may have been possible to safely advance
relfrozenxid by many more XIDs, at a relatively small cost in freezing
(possibly no extra cost at all) -- but VACUUM rigidly coupled freezing
with advancing relfrozenxid, missing all this.

Teach VACUUM to track the newest possible safe final relfrozenxid
dynamically (and to track a new value for relminmxid).  In the extreme
though common case where all tuples are already frozen, or became frozen
(or were removed by pruning), the final relfrozenxid value will be
exactly equal to the OldestXmin value used by the same VACUUM operation.
A later patch will overhaul the strategy that VACUUM uses for freezing
so that relfrozenxid will tend to get set to a value that's relatively
close to OldestXmin in almost all cases.

Final relfrozenxid values still follow the same rules as before.  They
must still be >= FreezeLimit in an aggressive VACUUM.  Non-aggressive
VACUUMs can set relfrozenxid to any value that's greater than the
preexisting relfrozenxid, which could be either much earlier or much
later than FreezeLimit.  Much depends on workload characteristics.  In
practice there is significant natural variation that we can take
advantage of.

Credit for the general idea of using the oldest extant XID to set
pg_class.relfrozenxid at the end of VACUUM goes to Andres Freund.

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-WzkymFbz6D_vL+jmqSn_5q1wsFvFrE+37yLgL_Rkfd6Gzg@mail.gmail.com
---
 src/include/access/heapam.h          |   4 +-
 src/include/access/heapam_xlog.h     |   4 +-
 src/include/commands/vacuum.h        |   1 +
 src/backend/access/heap/heapam.c     | 186 ++++++++++++++++++++-------
 src/backend/access/heap/vacuumlazy.c |  86 +++++++++----
 src/backend/commands/cluster.c       |   5 +-
 src/backend/commands/vacuum.c        |  34 ++++-
 7 files changed, 241 insertions(+), 79 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 0ad87730e..d35402f9f 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -168,7 +168,9 @@ extern bool heap_freeze_tuple(HeapTupleHeader tuple,
 							  TransactionId relfrozenxid, TransactionId relminmxid,
 							  TransactionId cutoff_xid, TransactionId cutoff_multi);
 extern bool heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-									MultiXactId cutoff_multi, Buffer buf);
+									MultiXactId cutoff_multi,
+									TransactionId *NewRelfrozenxid,
+									MultiXactId *NewRelminmxid, Buffer buf);
 extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
 
 extern void simple_heap_insert(Relation relation, HeapTuple tup);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 5c47fdcec..ae55c90f7 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -410,7 +410,9 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 									  TransactionId cutoff_xid,
 									  TransactionId cutoff_multi,
 									  xl_heap_freeze_tuple *frz,
-									  bool *totally_frozen);
+									  bool *totally_frozen,
+									  TransactionId *NewRelfrozenxid,
+									  MultiXactId *NewRelminmxid);
 extern void heap_execute_freeze_tuple(HeapTupleHeader tuple,
 									  xl_heap_freeze_tuple *xlrec_tp);
 extern XLogRecPtr log_heap_visible(RelFileNode rnode, Buffer heap_buffer,
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f..ead88edda 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -291,6 +291,7 @@ extern bool vacuum_set_xid_limits(Relation rel,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
+								  MultiXactId *oldestMxact,
 								  TransactionId *freezeLimit,
 								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 98230aac4..d85a817ff 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6087,12 +6087,24 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
  * FRM_RETURN_IS_MULTI
  *		The return value is a new MultiXactId to set as new Xmax.
  *		(caller must obtain proper infomask bits using GetMultiXactIdHintBits)
+ *
+ * "NewRelfrozenxid" is an output value; it's used to maintain target new
+ * relfrozenxid for the relation.  It can be ignored unless "flags" contains
+ * either FRM_NOOP or FRM_RETURN_IS_MULTI, because we only handle multiXacts
+ * here.  This follows the general convention: only track XIDs that will still
+ * be in the table after the ongoing VACUUM finishes.  Note that it's up to
+ * caller to maintain this when the Xid return value is itself an Xid.
+ *
+ * Note that we cannot depend on xmin to maintain NewRelfrozenxid.  We need to
+ * push maintenance of NewRelfrozenxid down this far, since in general xmin
+ * might have been frozen by an earlier VACUUM operation, in which case our
+ * caller will not have factored-in xmin when maintaining NewRelfrozenxid.
  */
 static TransactionId
 FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 				  TransactionId relfrozenxid, TransactionId relminmxid,
 				  TransactionId cutoff_xid, MultiXactId cutoff_multi,
-				  uint16 *flags)
+				  uint16 *flags, TransactionId *NewRelfrozenxid)
 {
 	TransactionId xid = InvalidTransactionId;
 	int			i;
@@ -6104,6 +6116,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	bool		has_lockers;
 	TransactionId update_xid;
 	bool		update_committed;
+	TransactionId tempNewRelfrozenxid;
 
 	*flags = 0;
 
@@ -6198,13 +6211,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 
 	/* is there anything older than the cutoff? */
 	need_replace = false;
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	for (i = 0; i < nmembers; i++)
 	{
 		if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-		{
 			need_replace = true;
-			break;
-		}
+		if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+			tempNewRelfrozenxid = members[i].xid;
 	}
 
 	/*
@@ -6213,6 +6226,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 */
 	if (!need_replace)
 	{
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 		*flags |= FRM_NOOP;
 		pfree(members);
 		return InvalidTransactionId;
@@ -6222,6 +6236,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 * If the multi needs to be updated, figure out which members do we need
 	 * to keep.
 	 */
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	nnewmembers = 0;
 	newmembers = palloc(sizeof(MultiXactMember) * nmembers);
 	has_lockers = false;
@@ -6303,7 +6318,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			 * list.)
 			 */
 			if (TransactionIdIsValid(update_xid))
+			{
 				newmembers[nnewmembers++] = members[i];
+				if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+					tempNewRelfrozenxid = members[i].xid;
+			}
 		}
 		else
 		{
@@ -6313,6 +6332,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			{
 				/* running locker cannot possibly be older than the cutoff */
 				Assert(!TransactionIdPrecedes(members[i].xid, cutoff_xid));
+				Assert(!TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid));
 				newmembers[nnewmembers++] = members[i];
 				has_lockers = true;
 			}
@@ -6341,6 +6361,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		if (update_committed)
 			*flags |= FRM_MARK_COMMITTED;
 		xid = update_xid;
+		/* Caller manages NewRelfrozenxid directly when we return an XID */
 	}
 	else
 	{
@@ -6350,6 +6371,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		 */
 		xid = MultiXactIdCreateFromMembers(nnewmembers, newmembers);
 		*flags |= FRM_RETURN_IS_MULTI;
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 	}
 
 	pfree(newmembers);
@@ -6368,6 +6390,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
  * will be totally frozen after these operations are performed and false if
  * more freezing will eventually be required.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will actually go on to freeze as indicated by our *frz output, so
+ * any (xmin, xmax, xvac) XIDs that we indicate need to be frozen won't need
+ * to be counted here.  Values are valid lower bounds at the point that the
+ * ongoing VACUUM finishes.
+ *
  * Caller is responsible for setting the offset field, if appropriate.
  *
  * It is assumed that the caller has checked the tuple with
@@ -6392,7 +6421,9 @@ bool
 heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 						  TransactionId relfrozenxid, TransactionId relminmxid,
 						  TransactionId cutoff_xid, TransactionId cutoff_multi,
-						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p)
+						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p,
+						  TransactionId *NewRelfrozenxid,
+						  MultiXactId *NewRelminmxid)
 {
 	bool		changed = false;
 	bool		xmax_already_frozen = false;
@@ -6436,6 +6467,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->t_infomask |= HEAP_XMIN_FROZEN;
 			changed = true;
 		}
+		else if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+		{
+			/* won't be frozen, but older than current NewRelfrozenxid */
+			*NewRelfrozenxid = xid;
+		}
 	}
 
 	/*
@@ -6453,10 +6489,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 	{
 		TransactionId newxmax;
 		uint16		flags;
+		TransactionId temp = *NewRelfrozenxid;
 
 		newxmax = FreezeMultiXactId(xid, tuple->t_infomask,
 									relfrozenxid, relminmxid,
-									cutoff_xid, cutoff_multi, &flags);
+									cutoff_xid, cutoff_multi, &flags, &temp);
 
 		freeze_xmax = (flags & FRM_INVALIDATE_XMAX);
 
@@ -6474,6 +6511,24 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			if (flags & FRM_MARK_COMMITTED)
 				frz->t_infomask |= HEAP_XMAX_COMMITTED;
 			changed = true;
+
+			if (TransactionIdPrecedes(newxmax, *NewRelfrozenxid))
+			{
+				/* New xmax is an XID older than new NewRelfrozenxid */
+				*NewRelfrozenxid = newxmax;
+			}
+		}
+		else if (flags & FRM_NOOP)
+		{
+			/*
+			 * Changing nothing, so might have to ratchet back NewRelminmxid,
+			 * NewRelfrozenxid, or both together
+			 */
+			if (MultiXactIdIsValid(xid) &&
+				MultiXactIdPrecedes(xid, *NewRelminmxid))
+				*NewRelminmxid = xid;
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 		else if (flags & FRM_RETURN_IS_MULTI)
 		{
@@ -6495,6 +6550,13 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->xmax = newxmax;
 
 			changed = true;
+
+			/*
+			 * New multixact might have remaining XID older than
+			 * NewRelfrozenxid
+			 */
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 	}
 	else if (TransactionIdIsNormal(xid))
@@ -6522,7 +6584,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			freeze_xmax = true;
 		}
 		else
+		{
 			freeze_xmax = false;
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			{
+				/* won't be frozen, but older than current NewRelfrozenxid */
+				*NewRelfrozenxid = xid;
+			}
+		}
 	}
 	else if ((tuple->t_infomask & HEAP_XMAX_INVALID) ||
 			 !TransactionIdIsValid(HeapTupleHeaderGetRawXmax(tuple)))
@@ -6569,6 +6638,9 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 		 * was removed in PostgreSQL 9.0.  Note that if we were to respect
 		 * cutoff_xid here, we'd need to make surely to clear totally_frozen
 		 * when we skipped freezing on that basis.
+		 *
+		 * Since we always freeze here, NewRelfrozenxid doesn't need to be
+		 * maintained.
 		 */
 		if (TransactionIdIsNormal(xid))
 		{
@@ -6646,11 +6718,14 @@ heap_freeze_tuple(HeapTupleHeader tuple,
 	xl_heap_freeze_tuple frz;
 	bool		do_freeze;
 	bool		tuple_totally_frozen;
+	TransactionId NewRelfrozenxid = FirstNormalTransactionId;
+	MultiXactId NewRelminmxid = FirstMultiXactId;
 
 	do_freeze = heap_prepare_freeze_tuple(tuple,
 										  relfrozenxid, relminmxid,
 										  cutoff_xid, cutoff_multi,
-										  &frz, &tuple_totally_frozen);
+										  &frz, &tuple_totally_frozen,
+										  &NewRelfrozenxid, &NewRelminmxid);
 
 	/*
 	 * Note that because this is not a WAL-logged operation, we don't need to
@@ -7080,6 +7155,15 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  * Check to see whether any of the XID fields of a tuple (xmin, xmax, xvac)
  * are older than the specified cutoff XID or MultiXactId.  If so, return true.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will never freeze any of the XIDs from the tuple, even when we say
+ * that they should.  If caller opts to go with our recommendation to freeze,
+ * then it must account for the fact that it shouldn't trust how we've set
+ * NewRelfrozenxid/NewRelminmxid.  (In practice aggressive VACUUMs always take
+ * our recommendation because they must, and non-aggressive VACUUMs always opt
+ * to not freeze, preferring to ratchet back NewRelfrozenxid instead).
+ *
  * It doesn't matter whether the tuple is alive or dead, we are checking
  * to see if a tuple needs to be removed or frozen to avoid wraparound.
  *
@@ -7088,74 +7172,86 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  */
 bool
 heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-						MultiXactId cutoff_multi, Buffer buf)
+						MultiXactId cutoff_multi,
+						TransactionId *NewRelfrozenxid,
+						MultiXactId *NewRelminmxid, Buffer buf)
 {
 	TransactionId xid;
+	bool		needs_freeze = false;
 
 	xid = HeapTupleHeaderGetXmin(tuple);
-	if (TransactionIdIsNormal(xid) &&
-		TransactionIdPrecedes(xid, cutoff_xid))
-		return true;
+	if (TransactionIdIsNormal(xid))
+	{
+		if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			*NewRelfrozenxid = xid;
+		if (TransactionIdPrecedes(xid, cutoff_xid))
+			needs_freeze = true;
+	}
 
 	/*
 	 * The considerations for multixacts are complicated; look at
 	 * heap_prepare_freeze_tuple for justifications.  This routine had better
 	 * be in sync with that one!
+	 *
+	 * (Actually, we maintain NewRelminmxid differently here, because we
+	 * assume that XIDs that should be frozen according to cutoff_xid won't
+	 * be, whereas heap_prepare_freeze_tuple makes the opposite assumption.)
 	 */
 	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
 	{
 		MultiXactId multi;
+		MultiXactMember *members;
+		int			nmembers;
 
 		multi = HeapTupleHeaderGetRawXmax(tuple);
-		if (!MultiXactIdIsValid(multi))
-		{
-			/* no xmax set, ignore */
-			;
-		}
-		else if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
+		if (MultiXactIdIsValid(multi) &&
+			MultiXactIdPrecedes(multi, *NewRelminmxid))
+			*NewRelminmxid = multi;
+
+		if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
 			return true;
 		else if (MultiXactIdPrecedes(multi, cutoff_multi))
-			return true;
-		else
+			needs_freeze = true;
+
+		/* need to check whether any member of the mxact is too old */
+		nmembers = GetMultiXactIdMembers(multi, &members, false,
+										 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
+
+		for (int i = 0; i < nmembers; i++)
 		{
-			MultiXactMember *members;
-			int			nmembers;
-			int			i;
-
-			/* need to check whether any member of the mxact is too old */
-
-			nmembers = GetMultiXactIdMembers(multi, &members, false,
-											 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
-
-			for (i = 0; i < nmembers; i++)
-			{
-				if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-				{
-					pfree(members);
-					return true;
-				}
-			}
-			if (nmembers > 0)
-				pfree(members);
+			if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
+				needs_freeze = true;
+			if (TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
 		}
+		if (nmembers > 0)
+			pfree(members);
 	}
 	else
 	{
 		xid = HeapTupleHeaderGetRawXmax(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
 	if (tuple->t_infomask & HEAP_MOVED)
 	{
 		xid = HeapTupleHeaderGetXvac(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
-	return false;
+	return needs_freeze;
 }
 
 /*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d804e2553..9cc5742ad 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -171,8 +171,10 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
-	/* Are FreezeLimit/MultiXactCutoff still valid? */
-	bool		freeze_cutoffs_valid;
+
+	/* Track new pg_class.relfrozenxid/pg_class.relminmxid values */
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -329,6 +331,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
 
@@ -362,8 +365,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 									   params->freeze_table_age,
 									   params->multixact_freeze_min_age,
 									   params->multixact_freeze_table_age,
-									   &OldestXmin, &FreezeLimit,
-									   &MultiXactCutoff);
+									   &OldestXmin, &OldestMxact,
+									   &FreezeLimit, &MultiXactCutoff);
 
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
@@ -470,8 +473,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
-	/* Track if cutoffs became invalid (possible in !aggressive case only) */
-	vacrel->freeze_cutoffs_valid = true;
+
+	/* Initialize values used to advance relfrozenxid/relminmxid at the end */
+	vacrel->NewRelfrozenxid = OldestXmin;
+	vacrel->NewRelminmxid = OldestMxact;
 
 	/*
 	 * Call lazy_scan_heap to perform all required heap pruning, index
@@ -524,16 +529,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
 	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
 	 * provided we didn't skip any all-visible (not all-frozen) pages using
-	 * the visibility map, and assuming that we didn't fail to get a cleanup
-	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
-	 * MultiXactCutoff) established for VACUUM operation.
+	 * the visibility map.  A non-aggressive VACUUM might only be able to
+	 * advance relfrozenxid to an XID from before FreezeLimit (or a relminmxid
+	 * from before MultiXactCutoff) when it wasn't possible to freeze some
+	 * tuples due to our inability to acquire a cleanup lock, but the effect
+	 * is usually insignificant -- NewRelfrozenxid value still has a decent
+	 * chance of being much more recent that the existing relfrozenxid.
 	 *
 	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
 	 * the rel_pages used by lazy_scan_heap, which won't match when we
 	 * happened to truncate the relation afterwards.
 	 */
-	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
-		!vacrel->freeze_cutoffs_valid)
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
@@ -545,12 +552,23 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	}
 	else
 	{
-		/* Can safely advance relfrozen and relminmxid, too */
+		/*
+		 * Aggressive case is strictly required to advance relfrozenxid, at
+		 * least up to FreezeLimit (same applies with relminmxid and its
+		 * cutoff, MultiXactCutoff).  Assert that we got this right now.
+		 */
 		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
 			   orig_rel_pages);
+		Assert(!aggressive ||
+			   TransactionIdPrecedesOrEquals(FreezeLimit,
+											 vacrel->NewRelfrozenxid));
+		Assert(!aggressive ||
+			   MultiXactIdPrecedesOrEquals(MultiXactCutoff,
+										   vacrel->NewRelminmxid));
+
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff,
+							vacrel->NewRelfrozenxid, vacrel->NewRelminmxid,
 							&frozenxid_updated, &minmulti_updated, false);
 	}
 
@@ -655,17 +673,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 OldestXmin, diff);
 			if (frozenxid_updated)
 			{
-				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				diff = (int32) (vacrel->NewRelfrozenxid - vacrel->relfrozenxid);
 				appendStringInfo(&buf,
 								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
-								 FreezeLimit, diff);
+								 vacrel->NewRelfrozenxid, diff);
 			}
 			if (minmulti_updated)
 			{
-				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				diff = (int32) (vacrel->NewRelminmxid - vacrel->relminmxid);
 				appendStringInfo(&buf,
 								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
-								 MultiXactCutoff, diff);
+								 vacrel->NewRelminmxid, diff);
 			}
 			if (orig_rel_pages > 0)
 			{
@@ -1580,6 +1598,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	int			nfrozen;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1588,6 +1608,8 @@ lazy_scan_prune(LVRelState *vacrel,
 retry:
 
 	/* Initialize (or reset) page-level counters */
+	NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	NewRelminmxid = vacrel->NewRelminmxid;
 	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
@@ -1797,7 +1819,9 @@ retry:
 									  vacrel->FreezeLimit,
 									  vacrel->MultiXactCutoff,
 									  &frozen[nfrozen],
-									  &tuple_totally_frozen))
+									  &tuple_totally_frozen,
+									  &NewRelfrozenxid,
+									  &NewRelminmxid))
 		{
 			/* Will execute freeze below */
 			frozen[nfrozen++].offset = offnum;
@@ -1811,13 +1835,16 @@ retry:
 			prunestate->all_frozen = false;
 	}
 
+	vacrel->offnum = InvalidOffsetNumber;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
 	 * that remains and needs to be considered for freezing now (LP_UNUSED and
 	 * LP_REDIRECT items also remain, but are of no further interest to us).
 	 */
-	vacrel->offnum = InvalidOffsetNumber;
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
 
 	/*
 	 * Consider the need to freeze any items with tuple storage from the page
@@ -1978,6 +2005,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	MultiXactId NewRelminmxid = vacrel->NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -2024,7 +2053,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
 		if (heap_tuple_needs_freeze(tupleheader,
 									vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
+									vacrel->MultiXactCutoff,
+									&NewRelfrozenxid, &NewRelminmxid, buf))
 		{
 			if (vacrel->aggressive)
 			{
@@ -2034,10 +2064,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 			}
 
 			/*
-			 * Current non-aggressive VACUUM operation definitely won't be
-			 * able to advance relfrozenxid or relminmxid
+			 * A non-aggressive VACUUM doesn't have to wait on a cleanup lock
+			 * to ensure that it advances relfrozenxid to a sufficiently
+			 * recent XID that happens to be present on this page.  It can
+			 * just accept an older New/final relfrozenxid instead.  There is
+			 * a decent chance that the problem will go away naturally.
 			 */
-			vacrel->freeze_cutoffs_valid = false;
 		}
 
 		num_tuples++;
@@ -2087,6 +2119,14 @@ lazy_scan_noprune(LVRelState *vacrel,
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * We have committed to not freezing the tuples on this page (always
+	 * happens with a non-aggressive VACUUM), so make sure that the target
+	 * relfrozenxid/relminmxid values reflect the XIDs/MXIDs we encountered
+	 */
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
+
 	/*
 	 * Now save details of the LP_DEAD items from the page in vacrel (though
 	 * only when VACUUM uses two-pass strategy).
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 02a7e94bf..a7e988298 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -767,6 +767,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	TupleDesc	oldTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TupleDesc	newTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TransactionId OldestXmin;
+	MultiXactId oldestMxact;
 	TransactionId FreezeXid;
 	MultiXactId MultiXactCutoff;
 	bool		use_sort;
@@ -856,8 +857,8 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * Since we're going to rewrite the whole table anyway, there's no reason
 	 * not to be aggressive about this.
 	 */
-	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
+	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0, &OldestXmin, &oldestMxact,
+						  &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index d1cadf126..4f07e426e 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -950,10 +950,28 @@ get_all_vacuum_rels(int options)
  * The output parameters are:
  * - oldestXmin is the Xid below which tuples deleted by any xact (that
  *   committed) should be considered DEAD, not just RECENTLY_DEAD.
- * - freezeLimit is the Xid below which all Xids are replaced by
- *	 FrozenTransactionId during vacuum.
+ * - oldestMxact is the Mxid below which MultiXacts are definitely not
+ *   seen as visible by any running transaction.
+ * - freezeLimit is the Xid below which all Xids are definitely replaced by
+ *   FrozenTransactionId during aggressive vacuums.
  * - multiXactCutoff is the value below which all MultiXactIds are removed
  *   from Xmax.
+ *
+ * oldestXmin and oldestMxact can be thought of as the most recent values that
+ * can ever be passed to vac_update_relstats() as frozenxid and minmulti
+ * arguments.  These exact values will be used when no newer XIDs or
+ * MultiXacts remain in the heap relation (e.g., with an empty table).  It's
+ * typical for vacuumlazy.c caller to notice that older XIDs/Multixacts remain
+ * in the table, which will force it to use older value.  These older final
+ * values may not be any newer than the preexisting frozenxid/minmulti values
+ * from pg_class in extreme cases.  The final values are frequently fairly
+ * close to the optimal values that we give to vacuumlazy.c, though.
+ *
+ * An aggressive VACUUM always provides vac_update_relstats() arguments that
+ * are >= freezeLimit and >= multiXactCutoff.  A non-aggressive VACUUM may
+ * provide arguments that are either newer or older than freezeLimit and
+ * multiXactCutoff, or non-valid values (indicating that pg_class level
+ * cutoffs cannot be advanced at all).
  */
 bool
 vacuum_set_xid_limits(Relation rel,
@@ -962,6 +980,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_min_age,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
+					  MultiXactId *oldestMxact,
 					  TransactionId *freezeLimit,
 					  MultiXactId *multiXactCutoff)
 {
@@ -970,7 +989,6 @@ vacuum_set_xid_limits(Relation rel,
 	int			effective_multixact_freeze_max_age;
 	TransactionId limit;
 	TransactionId safeLimit;
-	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
 	int			freezetable;
@@ -1066,9 +1084,11 @@ vacuum_set_xid_limits(Relation rel,
 						 effective_multixact_freeze_max_age / 2);
 	Assert(mxid_freezemin >= 0);
 
+	/* Remember for caller */
+	*oldestMxact = GetOldestMultiXactId();
+
 	/* compute the cutoff multi, being careful to generate a valid value */
-	oldestMxact = GetOldestMultiXactId();
-	mxactLimit = oldestMxact - mxid_freezemin;
+	mxactLimit = *oldestMxact - mxid_freezemin;
 	if (mxactLimit < FirstMultiXactId)
 		mxactLimit = FirstMultiXactId;
 
@@ -1083,8 +1103,8 @@ vacuum_set_xid_limits(Relation rel,
 				(errmsg("oldest multixact is far in the past"),
 				 errhint("Close open transactions with multixacts soon to avoid wraparound problems.")));
 		/* Use the safe limit, unless an older mxact is still running */
-		if (MultiXactIdPrecedes(oldestMxact, safeMxactLimit))
-			mxactLimit = oldestMxact;
+		if (MultiXactIdPrecedes(*oldestMxact, safeMxactLimit))
+			mxactLimit = *oldestMxact;
 		else
 			mxactLimit = safeMxactLimit;
 	}
-- 
2.30.2

From bb7ae7d77b55902f3003b5dc8851bd3f949c1d09 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 13 Dec 2021 15:00:49 -0800
Subject: [PATCH v7 5/6] Make block-level characteristics drive freezing.

Teach VACUUM to freeze all of the tuples on a page whenever it notices
that it would otherwise mark the page all-visible, without also marking
it all-frozen.  VACUUM won't freeze _any_ tuples on the page unless
_all_ tuples (that remain after pruning) are all-visible.  It may
occasionally be necessary to freeze the page due to the presence of a
particularly old XID, from before VACUUM's FreezeLimit cutoff.  But the
FreezeLimit mechanism will seldom have any impact on which pages are
frozen anymore -- it is just a backstop now.

Freezing can now informally be thought of as something that takes place
at the level of an entire page, or not at all -- differences in XIDs
among tuples on the same page are not interesting, barring extreme
cases.  Freezing a page is now practically synonymous with setting the
page to all-visible in the visibility map, at least to users.

The main upside of the new approach to freezing is that it makes the
overhead of vacuuming much more predictable over time.  We avoid the
need for large balloon payments, since the system no longer accumulates
"freezing debt" that can only be paid off by anti-wraparound vacuuming.
This seems to have been particularly troublesome with append-only
tables, especially in the common case where XIDs from pages that are
marked all-visible for the first time are still fairly young (in
particular, not as old as indicated by VACUUM's vacuum_freeze_min_age
freezing cutoff).  Before now, nothing stopped these pages from being
set to all-visible (without also being set to all-frozen) the first time
they were reached by VACUUM, which meant that they just couldn't be
frozen until the next anti-wraparound VACUUM -- at which point the XIDs
from the unfrozen tuples might be much older than vacuum_freeze_min_age.
In summary, the old vacuum_freeze_min_age-based FreezeLimit cutoff could
not _reliably_ limit freezing debt unless the GUC was set to 0.

There is a virtuous cycle enabled by the new approach to freezing:
freezing more tuples earlier during non-aggressive VACUUMs allows us to
advance relfrozenxid eagerly, which buys time.  This creates every
opportunity for the workload to naturally generate enough dead tuples
(or newly inserted tuples) to make the autovacuum launcher launch a
non-aggressive autovacuum.  The overall effect is that most individual
tables no longer require _any_ anti-wraparound vacuum operations.  This
effect also owes much to the enhancement added by commit ?????, which
loosened the coupling between freezing and advancing relfrozenxid,
allowing VACUUM to precisely determine a new relfrozenxid.

It's still possible (and sometimes even likely) that VACUUM won't be
able to freeze a tuple with a somewhat older XID due only to a cleanup
lock not being immediately available.  It's even possible that some
VACUUM operations will fail to advance relfrozenxid by very many XIDs as
a consequence.  But the impact over time should be negligible.  The next
VACUUM operation for the table will effectively get a new opportunity to
freeze (or perhaps remove) the same tuple that was originally missed.
Once that happens, relfrozenxid will completely catch up. (Actually, one
could reasonably argue that we never really "fell behind" in the first
place -- the amount of freezing needed to significantly advance
relfrozenxid won't have measurably increased at any point.  A once-off
drop in the extent to which VACUUM can advance relfrozenxid is almost
certainly harmless noise.)

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-WzkymFbz6D_vL+jmqSn_5q1wsFvFrE+37yLgL_Rkfd6Gzg@mail.gmail.com
---
 src/backend/access/heap/vacuumlazy.c | 101 ++++++++++++++++++++++-----
 1 file changed, 84 insertions(+), 17 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 9cc5742ad..52cfb00ea 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -168,6 +168,7 @@ typedef struct LVRelState
 
 	/* VACUUM operation's cutoff for pruning */
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
@@ -199,6 +200,7 @@ typedef struct LVRelState
 	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber removed_pages;	/* # pages removed by relation truncation */
+	BlockNumber newly_frozen_pages; /* # pages with tuples frozen by us */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -353,12 +355,19 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 								  RelationGetRelid(rel));
 
 	/*
-	 * Get cutoffs that determine which tuples we need to freeze during the
-	 * VACUUM operation.
+	 * Determine if this is to be an aggressive VACUUM.  This will eventually
+	 * be required for any table where (for whatever reason) no non-aggressive
+	 * VACUUM ran to completion and advanced relfrozenxid below.  This became
+	 * much rarer when the strategy used to determine what to freeze was
+	 * taught to focus on freezing whole physical pages as the page was about
+	 * to be set all-visible (to avoid big cliffs during aggressive VACUUMs).
 	 *
-	 * Also determines if this is to be an aggressive VACUUM.  This will
-	 * eventually be required for any table where (for whatever reason) no
-	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
+	 * Also gets cutoffs that determine which tuples we should definitely
+	 * freeze.  If any one tuple is from before FreezeLimit, we will freeze
+	 * the whole page, even when we wouldn't otherwise freeze because the page
+	 * can't be set all-visible. (Actually, we won't freeze whatever the
+	 * tuples are that are individually not all visible on the page, since
+	 * that would render the tuples all-visible prematurely.)
 	 */
 	aggressive = vacuum_set_xid_limits(rel,
 									   params->freeze_min_age,
@@ -471,6 +480,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/* Set cutoffs for entire VACUUM */
 	vacrel->OldestXmin = OldestXmin;
+	vacrel->OldestMxact = OldestMxact;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
 
@@ -651,12 +661,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u scanned (%.2f%% of total)\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u scanned (%.2f%% of total), %u newly frozen (%.2f%% of total)\n"),
 							 vacrel->removed_pages,
 							 vacrel->rel_pages,
 							 vacrel->scanned_pages,
 							 orig_rel_pages == 0 ? 0 :
-							 100.0 * vacrel->scanned_pages / orig_rel_pages);
+							 100.0 * vacrel->scanned_pages / orig_rel_pages,
+							 vacrel->newly_frozen_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * vacrel->newly_frozen_pages / orig_rel_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
@@ -824,6 +837,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	vacrel->scanned_pages = 0;
 	vacrel->frozenskipped_pages = 0;
 	vacrel->removed_pages = 0;
+	vacrel->newly_frozen_pages = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -1025,7 +1039,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 			}
 
 			/*
-			 * Otherwise it must be an all-visible (and possibly even
+			 * Otherwise it must be an all-visible (and very likely
 			 * all-frozen) page that we decided to process regardless
 			 * (SKIP_PAGES_THRESHOLD must not have been crossed).
 			 */
@@ -1589,7 +1603,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	ItemId		itemid;
 	HeapTupleData tuple;
 	HTSV_Result res;
-	int			tuples_deleted,
+	int			tuples_deleted = 0,
 				lpdead_items,
 				recently_dead_tuples,
 				num_tuples,
@@ -1600,6 +1614,9 @@ lazy_scan_prune(LVRelState *vacrel,
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 	TransactionId NewRelfrozenxid;
 	MultiXactId NewRelminmxid;
+	TransactionId FreezeLimit = vacrel->FreezeLimit;
+	MultiXactId MultiXactCutoff = vacrel->MultiXactCutoff;
+	bool		freezeblk = false;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1610,7 +1627,6 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	NewRelfrozenxid = vacrel->NewRelfrozenxid;
 	NewRelminmxid = vacrel->NewRelminmxid;
-	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
 	num_tuples = 0;
@@ -1625,9 +1641,9 @@ retry:
 	 * lpdead_items's final value can be thought of as the number of tuples
 	 * that were deleted from indexes.
 	 */
-	tuples_deleted = heap_page_prune(rel, buf, vistest,
-									 InvalidTransactionId, 0, &nnewlpdead,
-									 &vacrel->offnum);
+	tuples_deleted += heap_page_prune(rel, buf, vistest,
+									  InvalidTransactionId, 0, &nnewlpdead,
+									  &vacrel->offnum);
 
 	/*
 	 * Now scan the page to collect LP_DEAD items and check for tuples
@@ -1678,11 +1694,16 @@ retry:
 		 * vacrel->nonempty_pages value) is inherently race-prone.  It must be
 		 * treated as advisory/unreliable, so we might as well be slightly
 		 * optimistic.
+		 *
+		 * We delay setting all_visible to false due to seeing an LP_DEAD
+		 * item.  We need to test "is the page all_visible if we just consider
+		 * remaining tuples with tuple storage?" below, when considering if we
+		 * should freeze the tuples on the page.  (all_visible will be set to
+		 * false for caller once we've decided on what to freeze.)
 		 */
 		if (ItemIdIsDead(itemid))
 		{
 			deadoffsets[lpdead_items++] = offnum;
-			prunestate->all_visible = false;
 			prunestate->has_lpdead_items = true;
 			continue;
 		}
@@ -1816,8 +1837,8 @@ retry:
 		if (heap_prepare_freeze_tuple(tuple.t_data,
 									  vacrel->relfrozenxid,
 									  vacrel->relminmxid,
-									  vacrel->FreezeLimit,
-									  vacrel->MultiXactCutoff,
+									  FreezeLimit,
+									  MultiXactCutoff,
 									  &frozen[nfrozen],
 									  &tuple_totally_frozen,
 									  &NewRelfrozenxid,
@@ -1837,6 +1858,50 @@ retry:
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * Freeze the whole page using OldestXmin (not FreezeLimit) as our cutoff
+	 * if the page is now eligible to be marked all_visible (barring any
+	 * LP_DEAD items) when the page is not already eligible to be marked
+	 * all_frozen.  We generally expect to freeze all of a block's tuples
+	 * together and at once, or none at all.  FreezeLimit is just a backstop
+	 * mechanism that makes sure that we don't overlook one or two older
+	 * tuples.
+	 *
+	 * For example, it's just about possible that successive VACUUM operations
+	 * will never quite manage to use the main block-level logic to freeze one
+	 * old tuple from a page where all other tuples are continually updated.
+	 * We should not be in any hurry to freeze such a tuple.  Even still, it's
+	 * better if we take care of it before an anti-wraparound VACUUM becomes
+	 * necessary -- that would mean that we'd have to wait for a cleanup lock
+	 * during the aggressive VACUUM, which has risks of its own.
+	 *
+	 * FIXME This code structure has been used for prototyping and testing the
+	 * algorithm, details of which have settled.  Code itself to be rewritten,
+	 * though.  It is backwards right now -- should be _starting_ with
+	 * OldestXmin (not FreezeLimit), since that's what happens at the
+	 * conceptual level.
+	 *
+	 * TODO Make vacuum_freeze_min_age GUC/reloption default -1, which will be
+	 * interpreted as "whatever autovacuum_freeze_max_age/2 is".  Idea is to
+	 * make FreezeLimit into a true backstop, and to do our best to avoid
+	 * waiting for a cleanup lock (always prefer to punt to the next VACUUM,
+	 * since we can advance relfrozenxid to the oldest XID on the page inside
+	 * lazy_scan_noprune).
+	 */
+	if (!freezeblk &&
+		((nfrozen > 0 && nfrozen < num_tuples) ||
+		 (prunestate->all_visible && !prunestate->all_frozen)))
+	{
+		freezeblk = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+		goto retry;
+	}
+
+	/* Time to define all_visible in a way that accounts for LP_DEAD items */
+	if (lpdead_items > 0)
+		prunestate->all_visible = false;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
@@ -1854,6 +1919,8 @@ retry:
 	{
 		Assert(prunestate->hastup);
 
+		vacrel->newly_frozen_pages++;
+
 		/*
 		 * At least one tuple with storage needs to be frozen -- execute that
 		 * now.
@@ -1882,7 +1949,7 @@ retry:
 		{
 			XLogRecPtr	recptr;
 
-			recptr = log_heap_freeze(vacrel->rel, buf, vacrel->FreezeLimit,
+			recptr = log_heap_freeze(vacrel->rel, buf, FreezeLimit,
 									 frozen, nfrozen);
 			PageSetLSN(page, recptr);
 		}
-- 
2.30.2

From b091b9051f64d28a376768dbbf17479dc1238f89 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 23 Jan 2022 21:10:38 -0800
Subject: [PATCH v7 6/6] Add all-visible FSM heuristic.

When recording free space in all-frozen page, record that the page has
zero free space when it has less than half BLCKSZ worth of space,
according to the traditional definition.  Otherwise record free space as
usual.

Making all-visible pages resistant to change like this can be thought of
as a form of hysteresis.  The page is given an opportunity to "settle"
and permanently stay in the same state when the tuples on the page will
never be updated or deleted.  But when they are updated or deleted, the
page can once again be used to store any tuple.  Over time, most pages
tend to settle permanently in many workloads, perhaps only on the second
or third attempt.
---
 src/backend/access/heap/vacuumlazy.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 52cfb00ea..5608a6e19 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1238,6 +1238,13 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 				 */
 				freespace = PageGetHeapFreeSpace(page);
 
+				/*
+				 * An all-visible page should not have its free space
+				 * available from FSM unless it's more than half empty
+				 */
+				if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+					freespace = 0;
+
 				UnlockReleaseBuffer(buf);
 				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
@@ -1375,6 +1382,13 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 		{
 			Size		freespace = PageGetHeapFreeSpace(page);
 
+			/*
+			 * An all-visible page should not have its free space available
+			 * from FSM unless it's more than half empty
+			 */
+			if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+				freespace = 0;
+
 			UnlockReleaseBuffer(buf);
 			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 		}
@@ -2549,6 +2563,13 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 		page = BufferGetPage(buf);
 		freespace = PageGetHeapFreeSpace(page);
 
+		/*
+		 * An all-visible page should not have its free space available from
+		 * FSM unless it's more than half empty
+		 */
+		if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+			freespace = 0;
+
 		UnlockReleaseBuffer(buf);
 		RecordPageWithFreeSpace(vacrel->rel, tblk, freespace);
 		vacuumed_pages++;
-- 
2.30.2

From ff523df6b910f4a51377c9ec75c616e64372850b Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 11 Dec 2021 17:39:45 -0800
Subject: [PATCH v7 3/6] Consolidate VACUUM xid cutoff logic.

Push the logic for determining whether or not any given VACUUM operation
will be aggressive down into vacuum_set_xid_limits().  This makes its
function signature significantly simpler.

This refactoring work will make it easier to set/return an "oldestMxact"
value the function's vacuumlazy.c caller in a later commit that teaches
VACUUM to intelligently set relfrozenxid and relminmxid to the oldest
real remain xid/MultiXactId.

A VACUUM operation's oldestMxact can be thought of as the MultiXactId
equivalent of its OldestXmin: just as OldestXmin is used as our initial
target relfrozenxid (which we'll ratchet back as the VACUUM progresses
and notices that it'll leave older XIDs in place), oldestMxact will be
our initial target MultiXactId (for a target MultiXactId that is itself
ratcheted back in the same way).
---
 src/include/commands/vacuum.h        |   6 +-
 src/backend/access/heap/vacuumlazy.c |  32 +++----
 src/backend/commands/cluster.c       |   3 +-
 src/backend/commands/vacuum.c        | 134 +++++++++++++--------------
 4 files changed, 79 insertions(+), 96 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index e5e548d6b..d64f6268f 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -286,15 +286,13 @@ extern void vac_update_relstats(Relation relation,
 								bool *frozenxid_updated,
 								bool *minmulti_updated,
 								bool in_outer_xact);
-extern void vacuum_set_xid_limits(Relation rel,
+extern bool vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
 								  TransactionId *freezeLimit,
-								  TransactionId *xidFullScanLimit,
-								  MultiXactId *multiXactCutoff,
-								  MultiXactId *mxactFullScanLimit);
+								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
 									  MultiXactId relminmxid);
 extern void vac_update_datfrozenxid(void);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 71378740c..d804e2553 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -322,8 +322,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
-	TransactionId xidFullScanLimit;
-	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
@@ -351,24 +349,22 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_progress_start_command(PROGRESS_COMMAND_VACUUM,
 								  RelationGetRelid(rel));
 
-	vacuum_set_xid_limits(rel,
-						  params->freeze_min_age,
-						  params->freeze_table_age,
-						  params->multixact_freeze_min_age,
-						  params->multixact_freeze_table_age,
-						  &OldestXmin, &FreezeLimit, &xidFullScanLimit,
-						  &MultiXactCutoff, &mxactFullScanLimit);
-
 	/*
-	 * We request an aggressive scan if the table's frozen Xid is now older
-	 * than or equal to the requested Xid full-table scan limit; or if the
-	 * table's minimum MultiXactId is older than or equal to the requested
-	 * mxid full-table scan limit; or if DISABLE_PAGE_SKIPPING was specified.
+	 * Get cutoffs that determine which tuples we need to freeze during the
+	 * VACUUM operation.
+	 *
+	 * Also determines if this is to be an aggressive VACUUM.  This will
+	 * eventually be required for any table where (for whatever reason) no
+	 * non-aggressive VACUUM ran to completion, and advanced relfrozenxid.
 	 */
-	aggressive = TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
-											   xidFullScanLimit);
-	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
-											  mxactFullScanLimit);
+	aggressive = vacuum_set_xid_limits(rel,
+									   params->freeze_min_age,
+									   params->freeze_table_age,
+									   params->multixact_freeze_min_age,
+									   params->multixact_freeze_table_age,
+									   &OldestXmin, &FreezeLimit,
+									   &MultiXactCutoff);
+
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
 	{
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2e8efe4f8..02a7e94bf 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -857,8 +857,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * not to be aggressive about this.
 	 */
 	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, NULL, &MultiXactCutoff,
-						  NULL);
+						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 37413dd43..d1cadf126 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -942,25 +942,20 @@ get_all_vacuum_rels(int options)
  *
  * Input parameters are the target relation, applicable freeze age settings.
  *
+ * Return value indicates whether caller should do an aggressive VACUUM or
+ * not.  This is a VACUUM that cannot skip any pages using the visibility map
+ * (except all-frozen pages), which is guaranteed to be able to advance
+ * relfrozenxid and relminmxid.
+ *
  * The output parameters are:
- * - oldestXmin is the cutoff value used to distinguish whether tuples are
- *	 DEAD or RECENTLY_DEAD (see HeapTupleSatisfiesVacuum).
+ * - oldestXmin is the Xid below which tuples deleted by any xact (that
+ *   committed) should be considered DEAD, not just RECENTLY_DEAD.
  * - freezeLimit is the Xid below which all Xids are replaced by
  *	 FrozenTransactionId during vacuum.
- * - xidFullScanLimit (computed from freeze_table_age parameter)
- *	 represents a minimum Xid value; a table whose relfrozenxid is older than
- *	 this will have a full-table vacuum applied to it, to freeze tuples across
- *	 the whole table.  Vacuuming a table younger than this value can use a
- *	 partial scan.
- * - multiXactCutoff is the value below which all MultiXactIds are removed from
- *	 Xmax.
- * - mxactFullScanLimit is a value against which a table's relminmxid value is
- *	 compared to produce a full-table vacuum, as with xidFullScanLimit.
- *
- * xidFullScanLimit and mxactFullScanLimit can be passed as NULL if caller is
- * not interested.
+ * - multiXactCutoff is the value below which all MultiXactIds are removed
+ *   from Xmax.
  */
-void
+bool
 vacuum_set_xid_limits(Relation rel,
 					  int freeze_min_age,
 					  int freeze_table_age,
@@ -968,9 +963,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
 					  TransactionId *freezeLimit,
-					  TransactionId *xidFullScanLimit,
-					  MultiXactId *multiXactCutoff,
-					  MultiXactId *mxactFullScanLimit)
+					  MultiXactId *multiXactCutoff)
 {
 	int			freezemin;
 	int			mxid_freezemin;
@@ -980,6 +973,7 @@ vacuum_set_xid_limits(Relation rel,
 	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
+	int			freezetable;
 
 	/*
 	 * We can always ignore processes running lazy vacuum.  This is because we
@@ -1097,64 +1091,60 @@ vacuum_set_xid_limits(Relation rel,
 
 	*multiXactCutoff = mxactLimit;
 
-	if (xidFullScanLimit != NULL)
-	{
-		int			freezetable;
+	/*
+	 * Done setting output parameters; just need to figure out if caller needs
+	 * to do an aggressive VACUUM or not.
+	 *
+	 * Determine the table freeze age to use: as specified by the caller, or
+	 * vacuum_freeze_table_age, but in any case not more than
+	 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
+	 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
+	 * before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_freeze_table_age;
+	freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		Assert(mxactFullScanLimit != NULL);
+	/*
+	 * Compute XID limit causing an aggressive vacuum, being careful not to
+	 * generate a "permanent" XID
+	 */
+	limit = ReadNextTransactionId() - freezetable;
+	if (!TransactionIdIsNormal(limit))
+		limit = FirstNormalTransactionId;
+	if (TransactionIdPrecedesOrEquals(rel->rd_rel->relfrozenxid,
+									  limit))
+		return true;
 
-		/*
-		 * Determine the table freeze age to use: as specified by the caller,
-		 * or vacuum_freeze_table_age, but in any case not more than
-		 * autovacuum_freeze_max_age * 0.95, so that if you have e.g nightly
-		 * VACUUM schedule, the nightly VACUUM gets a chance to freeze tuples
-		 * before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_freeze_table_age;
-		freezetable = Min(freezetable, autovacuum_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
+	/*
+	 * Similar to the above, determine the table freeze age to use for
+	 * multixacts: as specified by the caller, or
+	 * vacuum_multixact_freeze_table_age, but in any case not more than
+	 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have e.g.
+	 * nightly VACUUM schedule, the nightly VACUUM gets a chance to freeze
+	 * multixacts before anti-wraparound autovacuum is launched.
+	 */
+	freezetable = multixact_freeze_table_age;
+	if (freezetable < 0)
+		freezetable = vacuum_multixact_freeze_table_age;
+	freezetable = Min(freezetable,
+					  effective_multixact_freeze_max_age * 0.95);
+	Assert(freezetable >= 0);
 
-		/*
-		 * Compute XID limit causing a full-table vacuum, being careful not to
-		 * generate a "permanent" XID.
-		 */
-		limit = ReadNextTransactionId() - freezetable;
-		if (!TransactionIdIsNormal(limit))
-			limit = FirstNormalTransactionId;
+	/*
+	 * Compute MultiXact limit causing an aggressive vacuum, being careful to
+	 * generate a valid MultiXact value
+	 */
+	mxactLimit = ReadNextMultiXactId() - freezetable;
+	if (mxactLimit < FirstMultiXactId)
+		mxactLimit = FirstMultiXactId;
+	if (MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
+									mxactLimit))
+		return true;
 
-		*xidFullScanLimit = limit;
-
-		/*
-		 * Similar to the above, determine the table freeze age to use for
-		 * multixacts: as specified by the caller, or
-		 * vacuum_multixact_freeze_table_age, but in any case not more than
-		 * autovacuum_multixact_freeze_table_age * 0.95, so that if you have
-		 * e.g. nightly VACUUM schedule, the nightly VACUUM gets a chance to
-		 * freeze multixacts before anti-wraparound autovacuum is launched.
-		 */
-		freezetable = multixact_freeze_table_age;
-		if (freezetable < 0)
-			freezetable = vacuum_multixact_freeze_table_age;
-		freezetable = Min(freezetable,
-						  effective_multixact_freeze_max_age * 0.95);
-		Assert(freezetable >= 0);
-
-		/*
-		 * Compute MultiXact limit causing a full-table vacuum, being careful
-		 * to generate a valid MultiXact value.
-		 */
-		mxactLimit = ReadNextMultiXactId() - freezetable;
-		if (mxactLimit < FirstMultiXactId)
-			mxactLimit = FirstMultiXactId;
-
-		*mxactFullScanLimit = mxactLimit;
-	}
-	else
-	{
-		Assert(mxactFullScanLimit == NULL);
-	}
+	return false;
 }
 
 /*
-- 
2.30.2

From 18380fbc60810eeea2b60ba33e7a2850d0212fdd Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 21 Nov 2021 14:47:11 -0800
Subject: [PATCH v7 2/6] Add VACUUM instrumentation for scanned pages,
 relfrozenxid.

Report on scanned pages within VACUUM VERBOSE and autovacuum logging.
These are pages that were physically scanned during the VACUUM
operation, including pages that are marked all-visible in the visibility
map that were nevertheless scanned (typically because the visibility map
skipping decided not to skip a skippable page that is physically
surrounded by non-skippable pages).

Also report when relfrozenxid is advanced by VACUUM, and by how much.
Rename the user-visible OldestXmin output field to "removal cutoff", and
show some supplementary information: how far behind the cutoff is
(number of XIDs behind) by the time the VACUUM operation finished.  This
should give users some chance of figuring out what's _not_ working, and
highlights the relationship between OldestXmin and the relfrozenxid.

Finally, add instrumentation of "missed dead tuples", and the number of
pages that had at least one such tuple.  These are fully DEAD (not just
RECENTLY_DEAD) tuples with storage that could not be pruned due to an
inability to acquire a cleanup lock.  This is a replacement for the
"skipped due to pin" instrumentation removed by the previous commit.  It
shows more details than before for pages where failing to get a cleanup
lock actually mattered (those with missed dead tuples), and shows
nothing for pages where no real work was missed.  In practice there
seems to be far more of the latter than of the former, so the signal to
noise ratio is much improved.

Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/CAH2-Wznp=c=Opj8Z7RMR3G=ec3_JfGYMN_YvmCEjoPCHzWbx0g@mail.gmail.com
---
 src/include/commands/vacuum.h        |  2 +
 src/backend/access/heap/vacuumlazy.c | 97 +++++++++++++++++++---------
 src/backend/commands/analyze.c       |  3 +
 src/backend/commands/vacuum.c        |  9 +++
 4 files changed, 82 insertions(+), 29 deletions(-)

diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 5d0bdfa42..e5e548d6b 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -283,6 +283,8 @@ extern void vac_update_relstats(Relation relation,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
+								bool *frozenxid_updated,
+								bool *minmulti_updated,
 								bool in_outer_xact);
 extern void vacuum_set_xid_limits(Relation rel,
 								  int freeze_min_age, int freeze_table_age,
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index a9c83f6dc..71378740c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -198,6 +198,7 @@ typedef struct LVRelState
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber removed_pages;	/* # pages removed by relation truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
+	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
 
 	/* Statistics output by us, for table */
@@ -211,8 +212,8 @@ typedef struct LVRelState
 	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
-	int64		new_dead_tuples;	/* new estimated total # of dead items in
-									 * table */
+	int64		recently_dead_tuples;	/* # dead, but not yet removable */
+	int64		missed_dead_tuples; /* # removable, but not removed */
 	int64		num_tuples;		/* total number of nonremovable tuples */
 	int64		live_tuples;	/* live tuples (reltuples estimate) */
 } LVRelState;
@@ -317,6 +318,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				write_rate;
 	bool		aggressive,
 				skipwithvm;
+	bool		frozenxid_updated,
+				minmulti_updated;
 	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
@@ -538,9 +541,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	{
 		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
 		Assert(!aggressive);
+		frozenxid_updated = minmulti_updated = false;
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							InvalidTransactionId, InvalidMultiXactId, false);
+							InvalidTransactionId, InvalidMultiXactId,
+							NULL, NULL, false);
 	}
 	else
 	{
@@ -549,7 +554,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 			   orig_rel_pages);
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff, false);
+							FreezeLimit, MultiXactCutoff,
+							&frozenxid_updated, &minmulti_updated, false);
 	}
 
 	/*
@@ -565,7 +571,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	pgstat_report_vacuum(RelationGetRelid(rel),
 						 rel->rd_rel->relisshared,
 						 Max(new_live_tuples, 0),
-						 vacrel->new_dead_tuples);
+						 vacrel->recently_dead_tuples +
+						 vacrel->missed_dead_tuples);
 	pgstat_progress_end_command();
 
 	if (instrument)
@@ -578,6 +585,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
+			int32		diff;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -629,16 +637,40 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u scanned (%.2f%% of total)\n"),
 							 vacrel->removed_pages,
 							 vacrel->rel_pages,
-							 vacrel->frozenskipped_pages);
+							 vacrel->scanned_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * vacrel->scanned_pages / orig_rel_pages);
 			appendStringInfo(&buf,
-							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
+							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
 							 (long long) vacrel->new_rel_tuples,
-							 (long long) vacrel->new_dead_tuples,
-							 OldestXmin);
+							 (long long) vacrel->recently_dead_tuples);
+			if (vacrel->missed_dead_tuples > 0)
+				appendStringInfo(&buf,
+								 _("tuples missed: %lld dead from %u pages not removed due to cleanup lock contention\n"),
+								 (long long) vacrel->missed_dead_tuples,
+								 vacrel->missed_dead_pages);
+			diff = (int32) (ReadNextTransactionId() - OldestXmin);
+			appendStringInfo(&buf,
+							 _("removable cutoff: %u, older by %d xids when operation ended\n"),
+							 OldestXmin, diff);
+			if (frozenxid_updated)
+			{
+				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				appendStringInfo(&buf,
+								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
+								 FreezeLimit, diff);
+			}
+			if (minmulti_updated)
+			{
+				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				appendStringInfo(&buf,
+								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
+								 MultiXactCutoff, diff);
+			}
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -779,13 +811,15 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	vacrel->frozenskipped_pages = 0;
 	vacrel->removed_pages = 0;
 	vacrel->lpdead_item_pages = 0;
+	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
 
 	/* Initialize instrumentation counters */
 	vacrel->num_index_scans = 0;
 	vacrel->tuples_deleted = 0;
 	vacrel->lpdead_items = 0;
-	vacrel->new_dead_tuples = 0;
+	vacrel->recently_dead_tuples = 0;
+	vacrel->missed_dead_tuples = 0;
 	vacrel->num_tuples = 0;
 	vacrel->live_tuples = 0;
 
@@ -1334,7 +1368,8 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	 * (unlikely) scenario that new_live_tuples is -1, take it as zero.
 	 */
 	vacrel->new_rel_tuples =
-		Max(vacrel->new_live_tuples, 0) + vacrel->new_dead_tuples;
+		Max(vacrel->new_live_tuples, 0) + vacrel->recently_dead_tuples +
+		vacrel->missed_dead_tuples;
 
 	/*
 	 * Release any remaining pin on visibility map page.
@@ -1542,7 +1577,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				lpdead_items,
-				new_dead_tuples,
+				recently_dead_tuples,
 				num_tuples,
 				live_tuples;
 	int			nnewlpdead;
@@ -1559,7 +1594,7 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	tuples_deleted = 0;
 	lpdead_items = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
 	num_tuples = 0;
 	live_tuples = 0;
 
@@ -1718,11 +1753,11 @@ retry:
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * If tuple is recently deleted then we must not remove it
-				 * from relation.  (We only remove items that are LP_DEAD from
+				 * If tuple is recently dead then we must not remove it from
+				 * the relation.  (We only remove items that are LP_DEAD from
 				 * pruning.)
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				prunestate->all_visible = false;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
@@ -1898,7 +1933,7 @@ retry:
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
 	vacrel->lpdead_items += lpdead_items;
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
 }
@@ -1943,7 +1978,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	int			lpdead_items,
 				num_tuples,
 				live_tuples,
-				new_dead_tuples;
+				recently_dead_tuples,
+				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 
@@ -1955,7 +1991,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 	lpdead_items = 0;
 	num_tuples = 0;
 	live_tuples = 0;
-	new_dead_tuples = 0;
+	recently_dead_tuples = 0;
+	missed_dead_tuples = 0;
 
 	maxoff = PageGetMaxOffsetNumber(page);
 	for (offnum = FirstOffsetNumber;
@@ -2029,16 +2066,15 @@ lazy_scan_noprune(LVRelState *vacrel,
 				/*
 				 * There is some useful work for pruning to do, that won't be
 				 * done due to failure to get a cleanup lock.
-				 *
-				 * TODO Add dedicated instrumentation for this case
 				 */
+				missed_dead_tuples++;
 				break;
 			case HEAPTUPLE_RECENTLY_DEAD:
 
 				/*
-				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 * Count in recently_dead_tuples, just like lazy_scan_prune
 				 */
-				new_dead_tuples++;
+				recently_dead_tuples++;
 				break;
 			case HEAPTUPLE_INSERT_IN_PROGRESS:
 
@@ -2074,7 +2110,7 @@ lazy_scan_noprune(LVRelState *vacrel,
 			 */
 			*hastup = true;
 			num_tuples += lpdead_items;
-			/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+			missed_dead_tuples += lpdead_items;
 		}
 
 		/* Caller records free space, with or without LP_DEAD items */
@@ -2120,9 +2156,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 	/*
 	 * Finally, add relevant page-local counts to whole-VACUUM counts
 	 */
-	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->recently_dead_tuples += recently_dead_tuples;
+	vacrel->missed_dead_tuples += missed_dead_tuples;
 	vacrel->num_tuples += num_tuples;
 	vacrel->live_tuples += live_tuples;
+	if (missed_dead_tuples > 0)
+		vacrel->missed_dead_pages++;
 
 	/* Caller won't need to call lazy_scan_prune with same page */
 	return true;
@@ -2201,8 +2240,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * dead_items space is not CPU cache resident.
 		 *
 		 * We don't take any special steps to remember the LP_DEAD items (such
-		 * as counting them in new_dead_tuples report to the stats collector)
-		 * when the optimization is applied.  Though the accounting used in
+		 * as counting them in our final report to the stats collector) when
+		 * the optimization is applied.  Though the accounting used in
 		 * analyze.c's acquire_sample_rows() will recognize the same LP_DEAD
 		 * items as dead rows in its own stats collector report, that's okay.
 		 * The discrepancy should be negligible.  If this optimization is ever
@@ -3329,7 +3368,7 @@ update_index_statistics(LVRelState *vacrel)
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
-							false);
+							NULL, NULL, false);
 	}
 }
 
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index a0da998c2..736479295 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -645,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 
 		/* Same for indexes */
@@ -661,6 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
+								NULL, NULL,
 								in_outer_xact);
 		}
 	}
@@ -673,6 +675,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		vac_update_relstats(onerel, -1, totalrows,
 							0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
+							NULL, NULL,
 							in_outer_xact);
 	}
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index d1dadc54e..37413dd43 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1315,6 +1315,7 @@ vac_update_relstats(Relation relation,
 					BlockNumber num_all_visible_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
+					bool *frozenxid_updated, bool *minmulti_updated,
 					bool in_outer_xact)
 {
 	Oid			relid = RelationGetRelid(relation);
@@ -1390,22 +1391,30 @@ vac_update_relstats(Relation relation,
 	 * This should match vac_update_datfrozenxid() concerning what we consider
 	 * to be "in the future".
 	 */
+	if (frozenxid_updated)
+		*frozenxid_updated = false;
 	if (TransactionIdIsNormal(frozenxid) &&
 		pgcform->relfrozenxid != frozenxid &&
 		(TransactionIdPrecedes(pgcform->relfrozenxid, frozenxid) ||
 		 TransactionIdPrecedes(ReadNextTransactionId(),
 							   pgcform->relfrozenxid)))
 	{
+		if (frozenxid_updated)
+			*frozenxid_updated = true;
 		pgcform->relfrozenxid = frozenxid;
 		dirty = true;
 	}
 
 	/* Similarly for relminmxid */
+	if (minmulti_updated)
+		*minmulti_updated = false;
 	if (MultiXactIdIsValid(minmulti) &&
 		pgcform->relminmxid != minmulti &&
 		(MultiXactIdPrecedes(pgcform->relminmxid, minmulti) ||
 		 MultiXactIdPrecedes(ReadNextMultiXactId(), pgcform->relminmxid)))
 	{
+		if (minmulti_updated)
+			*minmulti_updated = true;
 		pgcform->relminmxid = minmulti;
 		dirty = true;
 	}
-- 
2.30.2

From 34a497a44a96aac5a5f2f35b52af7464d32e4d5b Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Wed, 17 Nov 2021 21:27:06 -0800
Subject: [PATCH v7 1/6] Simplify lazy_scan_heap's handling of scanned pages.

Redefine a scanned page as any heap page that actually gets pinned by
VACUUM's first pass over the heap.  Pages counted by scanned_pages are
now the complement of the pages that are skipped over using the
visibility map.  This new definition significantly simplifies quite a
few things.

Now heap relation truncation, visibility map bit setting, tuple counting
(e.g., for pg_class.reltuples), and tuple freezing all share a common
definition of scanned_pages.  That makes it possible to remove certain
special cases, that never made much sense.  We no longer need to track
tupcount_pages separately (see bugfix commit 1914c5ea for details),
since we now always count tuples from pages that are scanned_pages.  We
also don't need to needlessly distinguish between aggressive and
non-aggressive VACUUM operations when we cannot immediately acquire a
cleanup lock.

Since any VACUUM (not just an aggressive VACUUM) can sometimes advance
relfrozenxid, we now make non-aggressive VACUUMs work just a little
harder in order to make that desirable outcome more likely in practice.
Aggressive VACUUMs have long checked contended pages with only a shared
lock, to avoid needlessly waiting on a cleanup lock (in the common case
where the contended page has no tuples that need to be frozen anyway).
We still don't make non-aggressive VACUUMs wait for a cleanup lock, of
course -- if we did that they'd no longer be non-aggressive.  But we now
make the non-aggressive case notice that a failure to acquire a cleanup
lock on one particular heap page does not in itself make it unsafe to
advance relfrozenxid for the whole relation (which is what we usually
see in the aggressive case already).

We now also collect LP_DEAD items in the dead_items array in the case
where we cannot immediately get a cleanup lock on the buffer.  We cannot
prune without a cleanup lock, but opportunistic pruning may well have
left some LP_DEAD items behind in the past -- no reason to miss those.
Only VACUUM can mark these LP_DEAD items LP_UNUSED (no opportunistic
technique is independently capable of cleaning up line pointer bloat),
so we should not squander any opportunity to do that.  Commit 8523492d4e
taught VACUUM to set LP_DEAD line pointers to LP_UNUSED while only
holding an exclusive lock (not a cleanup lock), so we can expect to set
existing LP_DEAD items to LP_UNUSED reliably, even when we cannot
acquire our own cleanup lock at either pass over the heap (unless we opt
to skip index vacuuming, which implies that there is no second pass over
the heap).

We no longer report on "pin skipped pages" in log output.  A later patch
will add back an improved version of the same instrumentation.  We don't
want to show any information about any failures to acquire cleanup locks
unless we actually failed to do useful work as a consequence.  A page
that we could not acquire a cleanup lock on is now treated as equivalent
to any other scanned page in most cases.

Author: Peter Geoghegan <pg@bowt.ie>
Reviewed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Masahiko Sawada <sawada.mshk@gmail.com>
Discussion: https://postgr.es/m/CAH2-Wznp=c=Opj8Z7RMR3G=ec3_JfGYMN_YvmCEjoPCHzWbx0g@mail.gmail.com
---
 src/backend/access/heap/vacuumlazy.c          | 812 +++++++++++-------
 .../isolation/expected/vacuum-reltuples.out   |   2 +-
 .../isolation/specs/vacuum-reltuples.spec     |   7 +-
 3 files changed, 524 insertions(+), 297 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1c2f30b68..a9c83f6dc 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -143,6 +143,10 @@ typedef struct LVRelState
 	Relation   *indrels;
 	int			nindexes;
 
+	/* Aggressive VACUUM (scan all unfrozen pages)? */
+	bool		aggressive;
+	/* Use visibility map to skip? (disabled via reloption) */
+	bool		skipwithvm;
 	/* Wraparound failsafe has been triggered? */
 	bool		failsafe_active;
 	/* Consider index vacuuming bypass optimization? */
@@ -167,6 +171,8 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
+	/* Are FreezeLimit/MultiXactCutoff still valid? */
+	bool		freeze_cutoffs_valid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -188,10 +194,8 @@ typedef struct LVRelState
 	 */
 	VacDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
-	BlockNumber scanned_pages;	/* number of pages we examined */
-	BlockNumber pinskipped_pages;	/* # of pages skipped due to a pin */
-	BlockNumber frozenskipped_pages;	/* # of frozen pages we skipped */
-	BlockNumber tupcount_pages; /* # pages whose tuples we counted */
+	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
+	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber removed_pages;	/* # pages removed by relation truncation */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -204,6 +208,7 @@ typedef struct LVRelState
 
 	/* Instrumentation counters */
 	int			num_index_scans;
+	/* Counters that follow are only for scanned_pages */
 	int64		tuples_deleted; /* # deleted from table */
 	int64		lpdead_items;	/* # deleted from indexes */
 	int64		new_dead_tuples;	/* new estimated total # of dead items in
@@ -240,19 +245,22 @@ typedef struct LVSavedErrInfo
 
 
 /* non-export function prototypes */
-static void lazy_scan_heap(LVRelState *vacrel, VacuumParams *params,
-						   bool aggressive);
+static void lazy_scan_heap(LVRelState *vacrel, int nworkers);
+static bool lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf,
+								   BlockNumber blkno, Page page,
+								   bool sharelock, Buffer vmbuffer);
 static void lazy_scan_prune(LVRelState *vacrel, Buffer buf,
 							BlockNumber blkno, Page page,
 							GlobalVisState *vistest,
 							LVPagePruneState *prunestate);
+static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
+							  BlockNumber blkno, Page page,
+							  bool *hastup, bool *recordfreespace);
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
 static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
 								  Buffer buffer, int index, Buffer *vmbuffer);
-static bool lazy_check_needs_freeze(Buffer buf, bool *hastup,
-									LVRelState *vacrel);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void lazy_cleanup_all_indexes(LVRelState *vacrel);
 static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -307,16 +315,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	int			usecs;
 	double		read_rate,
 				write_rate;
-	bool		aggressive;		/* should we scan all unfrozen pages? */
-	bool		scanned_all_unfrozen;	/* actually scanned all such pages? */
+	bool		aggressive,
+				skipwithvm;
+	BlockNumber orig_rel_pages;
 	char	  **indnames = NULL;
 	TransactionId xidFullScanLimit;
 	MultiXactId mxactFullScanLimit;
 	BlockNumber new_rel_pages;
 	BlockNumber new_rel_allvisible;
 	double		new_live_tuples;
-	TransactionId new_frozen_xid;
-	MultiXactId new_min_multi;
 	ErrorContextCallback errcallback;
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
@@ -359,8 +366,16 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 											   xidFullScanLimit);
 	aggressive |= MultiXactIdPrecedesOrEquals(rel->rd_rel->relminmxid,
 											  mxactFullScanLimit);
+	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
+	{
+		/*
+		 * Force aggressive mode, and disable skipping blocks using the
+		 * visibility map (even those set all-frozen)
+		 */
 		aggressive = true;
+		skipwithvm = false;
+	}
 
 	/*
 	 * Setup error traceback support for ereport() first.  The idea is to set
@@ -423,6 +438,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	Assert(params->index_cleanup != VACOPTVALUE_UNSPECIFIED);
 	Assert(params->truncate != VACOPTVALUE_UNSPECIFIED &&
 		   params->truncate != VACOPTVALUE_AUTO);
+	vacrel->aggressive = aggressive;
+	vacrel->skipwithvm = skipwithvm;
 	vacrel->failsafe_active = false;
 	vacrel->consider_bypass_optimization = true;
 	vacrel->do_index_vacuuming = true;
@@ -454,35 +471,23 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
+	/* Track if cutoffs became invalid (possible in !aggressive case only) */
+	vacrel->freeze_cutoffs_valid = true;
 
 	/*
 	 * Call lazy_scan_heap to perform all required heap pruning, index
 	 * vacuuming, and heap vacuuming (plus related processing)
 	 */
-	lazy_scan_heap(vacrel, params, aggressive);
+	lazy_scan_heap(vacrel, params->nworkers);
 
 	/* Done with indexes */
 	vac_close_indexes(vacrel->nindexes, vacrel->indrels, NoLock);
 
 	/*
-	 * Compute whether we actually scanned the all unfrozen pages. If we did,
-	 * we can adjust relfrozenxid and relminmxid.
-	 *
-	 * NB: We need to check this before truncating the relation, because that
-	 * will change ->rel_pages.
-	 */
-	if ((vacrel->scanned_pages + vacrel->frozenskipped_pages)
-		< vacrel->rel_pages)
-	{
-		Assert(!aggressive);
-		scanned_all_unfrozen = false;
-	}
-	else
-		scanned_all_unfrozen = true;
-
-	/*
-	 * Optionally truncate the relation.
+	 * Optionally truncate the relation.  But remember the relation size used
+	 * by lazy_scan_prune for later first.
 	 */
+	orig_rel_pages = vacrel->rel_pages;
 	if (should_attempt_truncation(vacrel))
 	{
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_TRUNCATE,
@@ -508,28 +513,44 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 *
 	 * For safety, clamp relallvisible to be not more than what we're setting
 	 * relpages to.
-	 *
-	 * Also, don't change relfrozenxid/relminmxid if we skipped any pages,
-	 * since then we don't know for certain that all tuples have a newer xmin.
 	 */
-	new_rel_pages = vacrel->rel_pages;
+	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
 	new_live_tuples = vacrel->new_live_tuples;
 
 	visibilitymap_count(rel, &new_rel_allvisible, NULL);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
-	new_frozen_xid = scanned_all_unfrozen ? FreezeLimit : InvalidTransactionId;
-	new_min_multi = scanned_all_unfrozen ? MultiXactCutoff : InvalidMultiXactId;
-
-	vac_update_relstats(rel,
-						new_rel_pages,
-						new_live_tuples,
-						new_rel_allvisible,
-						vacrel->nindexes > 0,
-						new_frozen_xid,
-						new_min_multi,
-						false);
+	/*
+	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
+	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
+	 * provided we didn't skip any all-visible (not all-frozen) pages using
+	 * the visibility map, and assuming that we didn't fail to get a cleanup
+	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
+	 * MultiXactCutoff) established for VACUUM operation.
+	 *
+	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
+	 * the rel_pages used by lazy_scan_heap, which won't match when we
+	 * happened to truncate the relation afterwards.
+	 */
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
+		!vacrel->freeze_cutoffs_valid)
+	{
+		/* Cannot advance relfrozenxid/relminmxid -- just update pg_class */
+		Assert(!aggressive);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							InvalidTransactionId, InvalidMultiXactId, false);
+	}
+	else
+	{
+		/* Can safely advance relfrozen and relminmxid, too */
+		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
+			   orig_rel_pages);
+		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
+							new_rel_allvisible, vacrel->nindexes > 0,
+							FreezeLimit, MultiXactCutoff, false);
+	}
 
 	/*
 	 * Report results to the stats collector, too.
@@ -557,7 +578,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 		{
 			StringInfoData buf;
 			char	   *msgfmt;
-			BlockNumber orig_rel_pages;
 
 			TimestampDifference(starttime, endtime, &secs, &usecs);
 
@@ -609,10 +629,9 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped frozen\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped frozen\n"),
 							 vacrel->removed_pages,
 							 vacrel->rel_pages,
-							 vacrel->pinskipped_pages,
 							 vacrel->frozenskipped_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable, oldest xmin: %u\n"),
@@ -620,7 +639,6 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 (long long) vacrel->new_rel_tuples,
 							 (long long) vacrel->new_dead_tuples,
 							 OldestXmin);
-			orig_rel_pages = vacrel->rel_pages + vacrel->removed_pages;
 			if (orig_rel_pages > 0)
 			{
 				if (vacrel->do_index_vacuuming)
@@ -737,7 +755,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		supply.
  */
 static void
-lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
+lazy_scan_heap(LVRelState *vacrel, int nworkers)
 {
 	VacDeadItems *dead_items;
 	BlockNumber nblocks,
@@ -756,14 +774,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	GlobalVisState *vistest;
 
 	nblocks = RelationGetNumberOfBlocks(vacrel->rel);
-	next_unskippable_block = 0;
-	next_failsafe_block = 0;
-	next_fsm_block_to_vacuum = 0;
 	vacrel->rel_pages = nblocks;
 	vacrel->scanned_pages = 0;
-	vacrel->pinskipped_pages = 0;
 	vacrel->frozenskipped_pages = 0;
-	vacrel->tupcount_pages = 0;
 	vacrel->removed_pages = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -787,14 +800,16 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * dangerously old.
 	 */
 	lazy_check_wraparound_failsafe(vacrel);
+	next_failsafe_block = 0;
 
 	/*
 	 * Allocate the space for dead_items.  Note that this handles parallel
 	 * VACUUM initialization as part of allocating shared memory space used
 	 * for dead_items.
 	 */
-	dead_items_alloc(vacrel, params->nworkers);
+	dead_items_alloc(vacrel, nworkers);
 	dead_items = vacrel->dead_items;
+	next_fsm_block_to_vacuum = 0;
 
 	/* Report that we're scanning the heap, advertising total # of blocks */
 	initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
@@ -803,7 +818,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/*
-	 * Except when aggressive is set, we want to skip pages that are
+	 * Set things up for skipping blocks using visibility map.
+	 *
+	 * Except when vacrel->aggressive is set, we want to skip pages that are
 	 * all-visible according to the visibility map, but only when we can skip
 	 * at least SKIP_PAGES_THRESHOLD consecutive pages.  Since we're reading
 	 * sequentially, the OS should be doing readahead for us, so there's no
@@ -812,8 +829,8 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * page means that we can't update relfrozenxid, so we only want to do it
 	 * if we can skip a goodly number of pages.
 	 *
-	 * When aggressive is set, we can't skip pages just because they are
-	 * all-visible, but we can still skip pages that are all-frozen, since
+	 * When vacrel->aggressive is set, we can't skip pages just because they
+	 * are all-visible, but we can still skip pages that are all-frozen, since
 	 * such pages do not need freezing and do not affect the value that we can
 	 * safely set for relfrozenxid or relminmxid.
 	 *
@@ -836,17 +853,9 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 	 * just added to that page are necessarily newer than the GlobalXmin we
 	 * computed, so they'll have no effect on the value to which we can safely
 	 * set relfrozenxid.  A similar argument applies for MXIDs and relminmxid.
-	 *
-	 * We will scan the table's last page, at least to the extent of
-	 * determining whether it has tuples or not, even if it should be skipped
-	 * according to the above rules; except when we've already determined that
-	 * it's not worth trying to truncate the table.  This avoids having
-	 * lazy_truncate_heap() take access-exclusive lock on the table to attempt
-	 * a truncation that just fails immediately because there are tuples in
-	 * the last page.  This is worth avoiding mainly because such a lock must
-	 * be replayed on any hot standby, where it can be disruptive.
 	 */
-	if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+	next_unskippable_block = 0;
+	if (vacrel->skipwithvm)
 	{
 		while (next_unskippable_block < nblocks)
 		{
@@ -855,7 +864,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			vmstatus = visibilitymap_get_status(vacrel->rel,
 												next_unskippable_block,
 												&vmbuffer);
-			if (aggressive)
+			if (vacrel->aggressive)
 			{
 				if ((vmstatus & VISIBILITYMAP_ALL_FROZEN) == 0)
 					break;
@@ -882,13 +891,6 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		bool		all_visible_according_to_vm = false;
 		LVPagePruneState prunestate;
 
-		/*
-		 * Consider need to skip blocks.  See note above about forcing
-		 * scanning of last page.
-		 */
-#define FORCE_CHECK_PAGE() \
-		(blkno == nblocks - 1 && should_attempt_truncation(vacrel))
-
 		pgstat_progress_update_param(PROGRESS_VACUUM_HEAP_BLKS_SCANNED, blkno);
 
 		update_vacuum_error_info(vacrel, NULL, VACUUM_ERRCB_PHASE_SCAN_HEAP,
@@ -898,7 +900,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		{
 			/* Time to advance next_unskippable_block */
 			next_unskippable_block++;
-			if ((params->options & VACOPT_DISABLE_PAGE_SKIPPING) == 0)
+			if (vacrel->skipwithvm)
 			{
 				while (next_unskippable_block < nblocks)
 				{
@@ -907,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 					vmskipflags = visibilitymap_get_status(vacrel->rel,
 														   next_unskippable_block,
 														   &vmbuffer);
-					if (aggressive)
+					if (vacrel->aggressive)
 					{
 						if ((vmskipflags & VISIBILITYMAP_ALL_FROZEN) == 0)
 							break;
@@ -936,19 +938,27 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 			 * it's not all-visible.  But in an aggressive vacuum we know only
 			 * that it's not all-frozen, so it might still be all-visible.
 			 */
-			if (aggressive && VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
+			if (vacrel->aggressive &&
+				VM_ALL_VISIBLE(vacrel->rel, blkno, &vmbuffer))
 				all_visible_according_to_vm = true;
 		}
 		else
 		{
 			/*
-			 * The current block is potentially skippable; if we've seen a
-			 * long enough run of skippable blocks to justify skipping it, and
-			 * we're not forced to check it, then go ahead and skip.
-			 * Otherwise, the page must be at least all-visible if not
-			 * all-frozen, so we can set all_visible_according_to_vm = true.
+			 * The current page can be skipped if we've seen a long enough run
+			 * of skippable blocks to justify skipping it -- provided it's not
+			 * the last page in the relation (according to rel_pages/nblocks).
+			 *
+			 * We always scan the table's last page to determine whether it
+			 * has tuples or not, even if it would otherwise be skipped. This
+			 * avoids having lazy_truncate_heap() take access-exclusive lock
+			 * on the table to attempt a truncation that just fails
+			 * immediately because there are tuples on the last page.
+			 *
+			 * XXX Do we need to skip even the last block when every page in
+			 * the relation is all-visible?  We don't do that currently.
 			 */
-			if (skipping_blocks && !FORCE_CHECK_PAGE())
+			if (skipping_blocks && blkno < nblocks - 1)
 			{
 				/*
 				 * Tricky, tricky.  If this is in aggressive vacuum, the page
@@ -957,18 +967,32 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 				 * careful to count it as a skipped all-frozen page in that
 				 * case, or else we'll think we can't update relfrozenxid and
 				 * relminmxid.  If it's not an aggressive vacuum, we don't
-				 * know whether it was all-frozen, so we have to recheck; but
-				 * in this case an approximate answer is OK.
+				 * know whether it was initially all-frozen, so we have to
+				 * recheck.
 				 */
-				if (aggressive || VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
+				if (vacrel->aggressive ||
+					VM_ALL_FROZEN(vacrel->rel, blkno, &vmbuffer))
 					vacrel->frozenskipped_pages++;
 				continue;
 			}
+
+			/*
+			 * Otherwise it must be an all-visible (and possibly even
+			 * all-frozen) page that we decided to process regardless
+			 * (SKIP_PAGES_THRESHOLD must not have been crossed).
+			 */
 			all_visible_according_to_vm = true;
 		}
 
 		vacuum_delay_point();
 
+		/*
+		 * We're not skipping this page using the visibility map, and so it is
+		 * (by definition) a scanned page.  Any tuples from this page are now
+		 * guaranteed to be counted below, after some preparatory checks.
+		 */
+		vacrel->scanned_pages++;
+
 		/*
 		 * Regularly check if wraparound failsafe should trigger.
 		 *
@@ -1023,174 +1047,78 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		}
 
 		/*
-		 * Set up visibility map page as needed.
-		 *
 		 * Pin the visibility map page in case we need to mark the page
 		 * all-visible.  In most cases this will be very cheap, because we'll
-		 * already have the correct page pinned anyway.  However, it's
-		 * possible that (a) next_unskippable_block is covered by a different
-		 * VM page than the current block or (b) we released our pin and did a
-		 * cycle of index vacuuming.
+		 * already have the correct page pinned anyway.
 		 */
 		visibilitymap_pin(vacrel->rel, blkno, &vmbuffer);
 
+		/* Finished preparatory checks.  Actually scan the page. */
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno,
 								 RBM_NORMAL, vacrel->bstrategy);
+		page = BufferGetPage(buf);
 
 		/*
-		 * We need buffer cleanup lock so that we can prune HOT chains and
-		 * defragment the page.
+		 * We need a buffer cleanup lock to prune HOT chains and defragment
+		 * the page in lazy_scan_prune.  But when it's not possible to acquire
+		 * a cleanup lock right away, we may be able to settle for reduced
+		 * processing using lazy_scan_noprune.
 		 */
 		if (!ConditionalLockBufferForCleanup(buf))
 		{
-			bool		hastup;
+			bool		hastup,
+						recordfreespace;
 
-			/*
-			 * If we're not performing an aggressive scan to guard against XID
-			 * wraparound, and we don't want to forcibly check the page, then
-			 * it's OK to skip vacuuming pages we get a lock conflict on. They
-			 * will be dealt with in some future vacuum.
-			 */
-			if (!aggressive && !FORCE_CHECK_PAGE())
-			{
-				ReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
-				continue;
-			}
-
-			/*
-			 * Read the page with share lock to see if any xids on it need to
-			 * be frozen.  If not we just skip the page, after updating our
-			 * scan statistics.  If there are some, we wait for cleanup lock.
-			 *
-			 * We could defer the lock request further by remembering the page
-			 * and coming back to it later, or we could even register
-			 * ourselves for multiple buffers and then service whichever one
-			 * is received first.  For now, this seems good enough.
-			 *
-			 * If we get here with aggressive false, then we're just forcibly
-			 * checking the page, and so we don't want to insist on getting
-			 * the lock; we only need to know if the page contains tuples, so
-			 * that we can update nonempty_pages correctly.  It's convenient
-			 * to use lazy_check_needs_freeze() for both situations, though.
-			 */
 			LockBuffer(buf, BUFFER_LOCK_SHARE);
-			if (!lazy_check_needs_freeze(buf, &hastup, vacrel))
+
+			/* Check for new or empty pages before lazy_scan_noprune call */
+			if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, true,
+									   vmbuffer))
 			{
-				UnlockReleaseBuffer(buf);
-				vacrel->scanned_pages++;
-				vacrel->pinskipped_pages++;
-				if (hastup)
-					vacrel->nonempty_pages = blkno + 1;
+				/* Processed as new/empty page (lock and pin released) */
 				continue;
 			}
-			if (!aggressive)
+
+			/* Collect LP_DEAD items in dead_items array, count tuples */
+			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
+								  &recordfreespace))
 			{
+				Size		freespace;
+
 				/*
-				 * Here, we must not advance scanned_pages; that would amount
-				 * to claiming that the page contains no freezable tuples.
+				 * Processed page successfully (without cleanup lock) -- just
+				 * need to perform rel truncation and FSM steps, much like the
+				 * lazy_scan_prune case.  Don't bother trying to match its
+				 * visibility map setting steps, though.
 				 */
-				UnlockReleaseBuffer(buf);
-				vacrel->pinskipped_pages++;
 				if (hastup)
 					vacrel->nonempty_pages = blkno + 1;
+				if (recordfreespace)
+					freespace = PageGetHeapFreeSpace(page);
+				UnlockReleaseBuffer(buf);
+				if (recordfreespace)
+					RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
 			}
+
+			/*
+			 * lazy_scan_noprune could not do all required processing.  Wait
+			 * for a cleanup lock, and call lazy_scan_prune in the usual way.
+			 */
+			Assert(vacrel->aggressive);
 			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
 			LockBufferForCleanup(buf);
-			/* drop through to normal processing */
 		}
 
-		/*
-		 * By here we definitely have enough dead_items space for whatever
-		 * LP_DEAD tids are on this page, we have the visibility map page set
-		 * up in case we need to set this page's all_visible/all_frozen bit,
-		 * and we have a cleanup lock.  Any tuples on this page are now sure
-		 * to be "counted" by this VACUUM.
-		 *
-		 * One last piece of preamble needs to take place before we can prune:
-		 * we need to consider new and empty pages.
-		 */
-		vacrel->scanned_pages++;
-		vacrel->tupcount_pages++;
-
-		page = BufferGetPage(buf);
-
-		if (PageIsNew(page))
+		/* Check for new or empty pages before lazy_scan_prune call */
+		if (lazy_scan_new_or_empty(vacrel, buf, blkno, page, false, vmbuffer))
 		{
-			/*
-			 * All-zeroes pages can be left over if either a backend extends
-			 * the relation by a single page, but crashes before the newly
-			 * initialized page has been written out, or when bulk-extending
-			 * the relation (which creates a number of empty pages at the tail
-			 * end of the relation, but enters them into the FSM).
-			 *
-			 * Note we do not enter the page into the visibilitymap. That has
-			 * the downside that we repeatedly visit this page in subsequent
-			 * vacuums, but otherwise we'll never not discover the space on a
-			 * promoted standby. The harm of repeated checking ought to
-			 * normally not be too bad - the space usually should be used at
-			 * some point, otherwise there wouldn't be any regular vacuums.
-			 *
-			 * Make sure these pages are in the FSM, to ensure they can be
-			 * reused. Do that by testing if there's any space recorded for
-			 * the page. If not, enter it. We do so after releasing the lock
-			 * on the heap page, the FSM is approximate, after all.
-			 */
-			UnlockReleaseBuffer(buf);
-
-			if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
-			{
-				Size		freespace = BLCKSZ - SizeOfPageHeaderData;
-
-				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
-			}
-			continue;
-		}
-
-		if (PageIsEmpty(page))
-		{
-			Size		freespace = PageGetHeapFreeSpace(page);
-
-			/*
-			 * Empty pages are always all-visible and all-frozen (note that
-			 * the same is currently not true for new pages, see above).
-			 */
-			if (!PageIsAllVisible(page))
-			{
-				START_CRIT_SECTION();
-
-				/* mark buffer dirty before writing a WAL record */
-				MarkBufferDirty(buf);
-
-				/*
-				 * It's possible that another backend has extended the heap,
-				 * initialized the page, and then failed to WAL-log the page
-				 * due to an ERROR.  Since heap extension is not WAL-logged,
-				 * recovery might try to replay our record setting the page
-				 * all-visible and find that the page isn't initialized, which
-				 * will cause a PANIC.  To prevent that, check whether the
-				 * page has been previously WAL-logged, and if not, do that
-				 * now.
-				 */
-				if (RelationNeedsWAL(vacrel->rel) &&
-					PageGetLSN(page) == InvalidXLogRecPtr)
-					log_newpage_buffer(buf, true);
-
-				PageSetAllVisible(page);
-				visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
-								  vmbuffer, InvalidTransactionId,
-								  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
-				END_CRIT_SECTION();
-			}
-
-			UnlockReleaseBuffer(buf);
-			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+			/* Processed as new/empty page (lock and pin released) */
 			continue;
 		}
 
 		/*
-		 * Prune and freeze tuples.
+		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
 		 * dead_items array.  This includes LP_DEAD line pointers that we
@@ -1398,7 +1326,7 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 
 	/* now we can compute the new value for pg_class.reltuples */
 	vacrel->new_live_tuples = vac_estimate_reltuples(vacrel->rel, nblocks,
-													 vacrel->tupcount_pages,
+													 vacrel->scanned_pages,
 													 vacrel->live_tuples);
 
 	/*
@@ -1447,6 +1375,137 @@ lazy_scan_heap(LVRelState *vacrel, VacuumParams *params, bool aggressive)
 		update_index_statistics(vacrel);
 }
 
+/*
+ *	lazy_scan_new_or_empty() -- lazy_scan_heap() new/empty page handling.
+ *
+ * Must call here to handle both new and empty pages before calling
+ * lazy_scan_prune or lazy_scan_noprune, since they're not prepared to deal
+ * with new or empty pages.
+ *
+ * It's necessary to consider new pages as a special case, since the rules for
+ * maintaining the visibility map and FSM with empty pages are a little
+ * different (though new pages can be truncated based on the usual rules).
+ *
+ * Empty pages are not really a special case -- they're just heap pages that
+ * have no allocated tuples (including even LP_UNUSED items).  You might
+ * wonder why we need to handle them here all the same.  It's only necessary
+ * because of a corner-case involving a hard crash during heap relation
+ * extension.  If we ever make relation-extension crash safe, then it should
+ * no longer be necessary to deal with empty pages here (or new pages, for
+ * that matter).
+ *
+ * Caller must hold at least a shared lock.  We might need to escalate the
+ * lock in that case, so the type of lock caller holds needs to be specified
+ * using 'sharelock' argument.
+ *
+ * Returns false in common case where caller should go on to call
+ * lazy_scan_prune (or lazy_scan_noprune).  Otherwise returns true, indicating
+ * that lazy_scan_heap is done processing the page, releasing lock on caller's
+ * behalf.
+ */
+static bool
+lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
+					   Page page, bool sharelock, Buffer vmbuffer)
+{
+	Size		freespace;
+
+	if (PageIsNew(page))
+	{
+		/*
+		 * All-zeroes pages can be left over if either a backend extends the
+		 * relation by a single page, but crashes before the newly initialized
+		 * page has been written out, or when bulk-extending the relation
+		 * (which creates a number of empty pages at the tail end of the
+		 * relation), and then enters them into the FSM.
+		 *
+		 * Note we do not enter the page into the visibilitymap. That has the
+		 * downside that we repeatedly visit this page in subsequent vacuums,
+		 * but otherwise we'll never discover the space on a promoted standby.
+		 * The harm of repeated checking ought to normally not be too bad. The
+		 * space usually should be used at some point, otherwise there
+		 * wouldn't be any regular vacuums.
+		 *
+		 * Make sure these pages are in the FSM, to ensure they can be reused.
+		 * Do that by testing if there's any space recorded for the page. If
+		 * not, enter it. We do so after releasing the lock on the heap page,
+		 * the FSM is approximate, after all.
+		 */
+		UnlockReleaseBuffer(buf);
+
+		if (GetRecordedFreeSpace(vacrel->rel, blkno) == 0)
+		{
+			freespace = BLCKSZ - SizeOfPageHeaderData;
+
+			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		}
+
+		return true;
+	}
+
+	if (PageIsEmpty(page))
+	{
+		/*
+		 * It seems likely that caller will always be able to get a cleanup
+		 * lock on an empty page.  But don't take any chances -- escalate to
+		 * an exclusive lock (still don't need a cleanup lock, though).
+		 */
+		if (sharelock)
+		{
+			LockBuffer(buf, BUFFER_LOCK_UNLOCK);
+			LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
+
+			if (!PageIsEmpty(page))
+			{
+				/* page isn't new or empty -- keep lock and pin for now */
+				return false;
+			}
+		}
+		else
+		{
+			/* Already have a full cleanup lock (which is more than enough) */
+		}
+
+		/*
+		 * Unlike new pages, empty pages are always set all-visible and
+		 * all-frozen.
+		 */
+		if (!PageIsAllVisible(page))
+		{
+			START_CRIT_SECTION();
+
+			/* mark buffer dirty before writing a WAL record */
+			MarkBufferDirty(buf);
+
+			/*
+			 * It's possible that another backend has extended the heap,
+			 * initialized the page, and then failed to WAL-log the page due
+			 * to an ERROR.  Since heap extension is not WAL-logged, recovery
+			 * might try to replay our record setting the page all-visible and
+			 * find that the page isn't initialized, which will cause a PANIC.
+			 * To prevent that, check whether the page has been previously
+			 * WAL-logged, and if not, do that now.
+			 */
+			if (RelationNeedsWAL(vacrel->rel) &&
+				PageGetLSN(page) == InvalidXLogRecPtr)
+				log_newpage_buffer(buf, true);
+
+			PageSetAllVisible(page);
+			visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
+							  vmbuffer, InvalidTransactionId,
+							  VISIBILITYMAP_ALL_VISIBLE | VISIBILITYMAP_ALL_FROZEN);
+			END_CRIT_SECTION();
+		}
+
+		freespace = PageGetHeapFreeSpace(page);
+		UnlockReleaseBuffer(buf);
+		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
+		return true;
+	}
+
+	/* page isn't new or empty -- keep lock and pin */
+	return false;
+}
+
 /*
  *	lazy_scan_prune() -- lazy_scan_heap() pruning and freezing.
  *
@@ -1491,6 +1550,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
 	maxoff = PageGetMaxOffsetNumber(page);
 
 retry:
@@ -1553,10 +1614,9 @@ retry:
 		 * LP_DEAD items are processed outside of the loop.
 		 *
 		 * Note that we deliberately don't set hastup=true in the case of an
-		 * LP_DEAD item here, which is not how lazy_check_needs_freeze() or
-		 * count_nondeletable_pages() do it -- they only consider pages empty
-		 * when they only have LP_UNUSED items, which is important for
-		 * correctness.
+		 * LP_DEAD item here, which is not how count_nondeletable_pages() does
+		 * it -- it only considers pages empty/truncatable when they have no
+		 * items at all (except LP_UNUSED items).
 		 *
 		 * Our assumption is that any LP_DEAD items we encounter here will
 		 * become LP_UNUSED inside lazy_vacuum_heap_page() before we actually
@@ -1843,6 +1903,231 @@ retry:
 	vacrel->live_tuples += live_tuples;
 }
 
+/*
+ *	lazy_scan_noprune() -- lazy_scan_prune() without pruning or freezing
+ *
+ * Caller need only hold a pin and share lock on the buffer, unlike
+ * lazy_scan_prune, which requires a full cleanup lock.
+ *
+ * While pruning isn't performed here, we can at least collect existing
+ * LP_DEAD items into the dead_items array for removal from indexes.  It's
+ * quite possible that earlier opportunistic pruning left LP_DEAD items
+ * behind, and we shouldn't miss out on an opportunity to make them reusable
+ * (VACUUM alone is capable of cleaning up line pointer bloat like this).
+ * Note that we'll only require an exclusive lock (not a cleanup lock) later
+ * on when we set these LP_DEAD items to LP_UNUSED.
+ *
+ * Freezing isn't performed here either.  For aggressive VACUUM callers, we
+ * may return false to indicate that a full cleanup lock is required.  This is
+ * necessary because pruning requires a cleanup lock, and because VACUUM
+ * cannot freeze a page's tuples until after pruning takes place (freezing
+ * tuples effectively requires a cleanup lock, though we don't need a cleanup
+ * lock in lazy_vacuum_heap_page or in lazy_scan_new_or_empty to set a heap
+ * page all-frozen in the visibility map).  Returns true to indicate that all
+ * required processing has been performed.
+ *
+ * See lazy_scan_prune for an explanation of hastup return flag.
+ * recordfreespace flag instructs caller on whether or not it should do
+ * generic FSM processing for page.
+ */
+static bool
+lazy_scan_noprune(LVRelState *vacrel,
+				  Buffer buf,
+				  BlockNumber blkno,
+				  Page page,
+				  bool *hastup,
+				  bool *recordfreespace)
+{
+	OffsetNumber offnum,
+				maxoff;
+	int			lpdead_items,
+				num_tuples,
+				live_tuples,
+				new_dead_tuples;
+	HeapTupleHeader tupleheader;
+	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+
+	Assert(BufferGetBlockNumber(buf) == blkno);
+
+	*hastup = false;			/* for now */
+	*recordfreespace = false;	/* for now */
+
+	lpdead_items = 0;
+	num_tuples = 0;
+	live_tuples = 0;
+	new_dead_tuples = 0;
+
+	maxoff = PageGetMaxOffsetNumber(page);
+	for (offnum = FirstOffsetNumber;
+		 offnum <= maxoff;
+		 offnum = OffsetNumberNext(offnum))
+	{
+		ItemId		itemid;
+		HeapTupleData tuple;
+
+		vacrel->offnum = offnum;
+		itemid = PageGetItemId(page, offnum);
+
+		if (!ItemIdIsUsed(itemid))
+			continue;
+
+		if (ItemIdIsRedirected(itemid))
+		{
+			*hastup = true;
+			continue;
+		}
+
+		if (ItemIdIsDead(itemid))
+		{
+			/*
+			 * Deliberately don't set hastup=true here.  See same point in
+			 * lazy_scan_prune for an explanation.
+			 */
+			deadoffsets[lpdead_items++] = offnum;
+			continue;
+		}
+
+		*hastup = true;			/* page prevents rel truncation */
+		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
+		if (heap_tuple_needs_freeze(tupleheader,
+									vacrel->FreezeLimit,
+									vacrel->MultiXactCutoff, buf))
+		{
+			if (vacrel->aggressive)
+			{
+				/* Going to have to get cleanup lock for lazy_scan_prune */
+				vacrel->offnum = InvalidOffsetNumber;
+				return false;
+			}
+
+			/*
+			 * Current non-aggressive VACUUM operation definitely won't be
+			 * able to advance relfrozenxid or relminmxid
+			 */
+			vacrel->freeze_cutoffs_valid = false;
+		}
+
+		num_tuples++;
+		ItemPointerSet(&(tuple.t_self), blkno, offnum);
+		tuple.t_data = (HeapTupleHeader) PageGetItem(page, itemid);
+		tuple.t_len = ItemIdGetLength(itemid);
+		tuple.t_tableOid = RelationGetRelid(vacrel->rel);
+
+		switch (HeapTupleSatisfiesVacuum(&tuple, vacrel->OldestXmin, buf))
+		{
+			case HEAPTUPLE_DELETE_IN_PROGRESS:
+			case HEAPTUPLE_LIVE:
+
+				/*
+				 * Count both cases as live, just like lazy_scan_prune
+				 */
+				live_tuples++;
+
+				break;
+			case HEAPTUPLE_DEAD:
+
+				/*
+				 * There is some useful work for pruning to do, that won't be
+				 * done due to failure to get a cleanup lock.
+				 *
+				 * TODO Add dedicated instrumentation for this case
+				 */
+				break;
+			case HEAPTUPLE_RECENTLY_DEAD:
+
+				/*
+				 * Count in new_dead_tuples, just like lazy_scan_prune
+				 */
+				new_dead_tuples++;
+				break;
+			case HEAPTUPLE_INSERT_IN_PROGRESS:
+
+				/*
+				 * Do not count these rows as live, just like lazy_scan_prune
+				 */
+				break;
+			default:
+				elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
+				break;
+		}
+
+	}
+
+	vacrel->offnum = InvalidOffsetNumber;
+
+	/*
+	 * Now save details of the LP_DEAD items from the page in vacrel (though
+	 * only when VACUUM uses two-pass strategy).
+	 */
+	if (vacrel->nindexes == 0)
+	{
+		/* Using one-pass strategy (since table has no indexes) */
+		if (lpdead_items > 0)
+		{
+			/*
+			 * Perfunctory handling for the corner case where a single pass
+			 * strategy VACUUM cannot get a cleanup lock, and it turns out
+			 * that there is one or more LP_DEAD items: just count the LP_DEAD
+			 * items as missed_dead_tuples instead. (This is a bit dishonest,
+			 * but it beats having to maintain specialized heap vacuuming code
+			 * forever, for vanishingly little benefit.)
+			 */
+			*hastup = true;
+			num_tuples += lpdead_items;
+			/* TODO HEAPTUPLE_DEAD style instrumentation needed here, too */
+		}
+
+		/* Caller records free space, with or without LP_DEAD items */
+		*recordfreespace = true;
+	}
+	else if (lpdead_items > 0)
+	{
+		VacDeadItems *dead_items = vacrel->dead_items;
+		ItemPointerData tmp;
+
+		/*
+		 * Page has LP_DEAD items, and so any references/TIDs that remain in
+		 * indexes will be deleted during index vacuuming (and then marked
+		 * LP_UNUSED in the heap).
+		 *
+		 * Don't record free space now -- leave it until second heap pass.
+		 */
+		vacrel->lpdead_item_pages++;
+
+		ItemPointerSetBlockNumber(&tmp, blkno);
+
+		for (int i = 0; i < lpdead_items; i++)
+		{
+			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
+			dead_items->items[dead_items->num_items++] = tmp;
+		}
+
+		Assert(dead_items->num_items <= dead_items->max_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
+									 dead_items->num_items);
+
+		vacrel->lpdead_items += lpdead_items;
+	}
+	else
+	{
+		/*
+		 * Caller won't be vacuuming this page later, so tell it to record
+		 * page's freespace in the FSM now, even though we didn't prune it
+		 */
+		*recordfreespace = true;
+	}
+
+	/*
+	 * Finally, add relevant page-local counts to whole-VACUUM counts
+	 */
+	vacrel->new_dead_tuples += new_dead_tuples;
+	vacrel->num_tuples += num_tuples;
+	vacrel->live_tuples += live_tuples;
+
+	/* Caller won't need to call lazy_scan_prune with same page */
+	return true;
+}
+
 /*
  * Main entry point for index vacuuming and heap vacuuming.
  *
@@ -2286,67 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 	return index;
 }
 
-/*
- *	lazy_check_needs_freeze() -- scan page to see if any tuples
- *					 need to be cleaned to avoid wraparound
- *
- * Returns true if the page needs to be vacuumed using cleanup lock.
- * Also returns a flag indicating whether page contains any tuples at all.
- */
-static bool
-lazy_check_needs_freeze(Buffer buf, bool *hastup, LVRelState *vacrel)
-{
-	Page		page = BufferGetPage(buf);
-	OffsetNumber offnum,
-				maxoff;
-	HeapTupleHeader tupleheader;
-
-	*hastup = false;
-
-	/*
-	 * New and empty pages, obviously, don't contain tuples. We could make
-	 * sure that the page is registered in the FSM, but it doesn't seem worth
-	 * waiting for a cleanup lock just for that, especially because it's
-	 * likely that the pin holder will do so.
-	 */
-	if (PageIsNew(page) || PageIsEmpty(page))
-		return false;
-
-	maxoff = PageGetMaxOffsetNumber(page);
-	for (offnum = FirstOffsetNumber;
-		 offnum <= maxoff;
-		 offnum = OffsetNumberNext(offnum))
-	{
-		ItemId		itemid;
-
-		/*
-		 * Set the offset number so that we can display it along with any
-		 * error that occurred while processing this tuple.
-		 */
-		vacrel->offnum = offnum;
-		itemid = PageGetItemId(page, offnum);
-
-		/* this should match hastup test in count_nondeletable_pages() */
-		if (ItemIdIsUsed(itemid))
-			*hastup = true;
-
-		/* dead and redirect items never need freezing */
-		if (!ItemIdIsNormal(itemid))
-			continue;
-
-		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
-
-		if (heap_tuple_needs_freeze(tupleheader, vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
-			break;
-	}							/* scan along page */
-
-	/* Clear the offset information once we have processed the given page. */
-	vacrel->offnum = InvalidOffsetNumber;
-
-	return (offnum <= maxoff);
-}
-
 /*
  * Trigger the failsafe to avoid wraparound failure when vacrel table has a
  * relfrozenxid and/or relminmxid that is dangerously far in the past.
@@ -2412,7 +2636,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 	{
 		double		reltuples = vacrel->new_rel_tuples;
 		bool		estimated_count =
-		vacrel->tupcount_pages < vacrel->rel_pages;
+		vacrel->scanned_pages < vacrel->rel_pages;
 
 		for (int idx = 0; idx < vacrel->nindexes; idx++)
 		{
@@ -2429,7 +2653,7 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
 		/* Outsource everything to parallel variant */
 		parallel_vacuum_cleanup_all_indexes(vacrel->pvs, vacrel->new_rel_tuples,
 											vacrel->num_index_scans,
-											(vacrel->tupcount_pages < vacrel->rel_pages));
+											(vacrel->scanned_pages < vacrel->rel_pages));
 	}
 }
 
@@ -2536,7 +2760,9 @@ lazy_cleanup_one_index(Relation indrel, IndexBulkDeleteResult *istat,
  * should_attempt_truncation - should we attempt to truncate the heap?
  *
  * Don't even think about it unless we have a shot at releasing a goodly
- * number of pages.  Otherwise, the time taken isn't worth it.
+ * number of pages.  Otherwise, the time taken isn't worth it, mainly because
+ * an AccessExclusive lock must be replayed on any hot standby, where it can
+ * be particularly disruptive.
  *
  * Also don't attempt it if wraparound failsafe is in effect.  It's hard to
  * predict how long lazy_truncate_heap will take.  Don't take any chances.
diff --git a/src/test/isolation/expected/vacuum-reltuples.out b/src/test/isolation/expected/vacuum-reltuples.out
index cdbe7f3a6..ce55376e7 100644
--- a/src/test/isolation/expected/vacuum-reltuples.out
+++ b/src/test/isolation/expected/vacuum-reltuples.out
@@ -45,7 +45,7 @@ step stats:
 
 relpages|reltuples
 --------+---------
-       1|       20
+       1|       21
 (1 row)
 
 
diff --git a/src/test/isolation/specs/vacuum-reltuples.spec b/src/test/isolation/specs/vacuum-reltuples.spec
index ae2f79b8f..a2a461f2f 100644
--- a/src/test/isolation/specs/vacuum-reltuples.spec
+++ b/src/test/isolation/specs/vacuum-reltuples.spec
@@ -2,9 +2,10 @@
 # to page pins. We absolutely need to avoid setting reltuples=0 in
 # such cases, since that interferes badly with planning.
 #
-# Expected result in second permutation is 20 tuples rather than 21 as
-# for the others, because vacuum should leave the previous result
-# (from before the insert) in place.
+# Expected result for all three permutation is 21 tuples, including
+# the second permutation.  VACUUM is able to count the concurrently
+# inserted tuple in its final reltuples, even when a cleanup lock
+# cannot be acquired on the affected heap page.
 
 setup {
     create table smalltbl
-- 
2.30.2

From 41136d2a8af434a095ce3e6dfdfbe4b48b9ec338 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sun, 23 Jan 2022 21:10:38 -0800
Subject: [PATCH v8 3/3] Add all-visible FSM heuristic.

When recording free space in all-frozen page, record that the page has
zero free space when it has less than half BLCKSZ worth of space,
according to the traditional definition.  Otherwise record free space as
usual.

Making all-visible pages resistant to change like this can be thought of
as a form of hysteresis.  The page is given an opportunity to "settle"
and permanently stay in the same state when the tuples on the page will
never be updated or deleted.  But when they are updated or deleted, the
page can once again be used to store any tuple.  Over time, most pages
tend to settle permanently in many workloads, perhaps only on the second
or third attempt.
---
 src/backend/access/heap/vacuumlazy.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index ea4b75189..95049ed25 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -1231,6 +1231,13 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 				 */
 				freespace = PageGetHeapFreeSpace(page);
 
+				/*
+				 * An all-visible page should not have its free space
+				 * available from FSM unless it's more than half empty
+				 */
+				if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+					freespace = 0;
+
 				UnlockReleaseBuffer(buf);
 				RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 				continue;
@@ -1368,6 +1375,13 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 		{
 			Size		freespace = PageGetHeapFreeSpace(page);
 
+			/*
+			 * An all-visible page should not have its free space available
+			 * from FSM unless it's more than half empty
+			 */
+			if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+				freespace = 0;
+
 			UnlockReleaseBuffer(buf);
 			RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 		}
@@ -2537,6 +2551,13 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 		page = BufferGetPage(buf);
 		freespace = PageGetHeapFreeSpace(page);
 
+		/*
+		 * An all-visible page should not have its free space available from
+		 * FSM unless it's more than half empty
+		 */
+		if (PageIsAllVisible(page) && freespace < BLCKSZ / 2)
+			freespace = 0;
+
 		UnlockReleaseBuffer(buf);
 		RecordPageWithFreeSpace(vacrel->rel, tblk, freespace);
 		vacuumed_pages++;
-- 
2.30.2

From 4838bd1f11b748d2082caedfe4da506b8fe3f67a Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 13 Dec 2021 15:00:49 -0800
Subject: [PATCH v8 2/3] Make block-level characteristics drive freezing.

Teach VACUUM to freeze all of the tuples on a page whenever it notices
that it would otherwise mark the page all-visible, without also marking
it all-frozen.  VACUUM won't freeze _any_ tuples on the page unless
_all_ tuples (that remain after pruning) are all-visible.  It may
occasionally be necessary to freeze the page due to the presence of a
particularly old XID, from before VACUUM's FreezeLimit cutoff.  But the
FreezeLimit mechanism will seldom have any impact on which pages are
frozen anymore -- it is just a backstop now.

Freezing can now informally be thought of as something that takes place
at the level of an entire page, or not at all -- differences in XIDs
among tuples on the same page are not interesting, barring extreme
cases.  Freezing a page is now practically synonymous with setting the
page to all-visible in the visibility map, at least to users.

The main upside of the new approach to freezing is that it makes the
overhead of vacuuming much more predictable over time.  We avoid the
need for large balloon payments, since the system no longer accumulates
"freezing debt" that can only be paid off by anti-wraparound vacuuming.
This seems to have been particularly troublesome with append-only
tables, especially in the common case where XIDs from pages that are
marked all-visible for the first time are still fairly young (in
particular, not as old as indicated by VACUUM's vacuum_freeze_min_age
freezing cutoff).  Before now, nothing stopped these pages from being
set to all-visible (without also being set to all-frozen) the first time
they were reached by VACUUM, which meant that they just couldn't be
frozen until the next anti-wraparound VACUUM -- at which point the XIDs
from the unfrozen tuples might be much older than vacuum_freeze_min_age.
In summary, the old vacuum_freeze_min_age-based FreezeLimit cutoff could
not _reliably_ limit freezing debt unless the GUC was set to 0.

There is a virtuous cycle enabled by the new approach to freezing:
freezing more tuples earlier during non-aggressive VACUUMs allows us to
advance relfrozenxid eagerly, which buys time.  This creates every
opportunity for the workload to naturally generate enough dead tuples
(or newly inserted tuples) to make the autovacuum launcher launch a
non-aggressive autovacuum.  The overall effect is that most individual
tables no longer require _any_ anti-wraparound vacuum operations.  This
effect also owes much to the enhancement added by commit ?????, which
loosened the coupling between freezing and advancing relfrozenxid,
allowing VACUUM to precisely determine a new relfrozenxid.

It's still possible (and sometimes even likely) that VACUUM won't be
able to freeze a tuple with a somewhat older XID due only to a cleanup
lock not being immediately available.  It's even possible that some
VACUUM operations will fail to advance relfrozenxid by very many XIDs as
a consequence.  But the impact over time should be negligible.  The next
VACUUM operation for the table will effectively get a new opportunity to
freeze (or perhaps remove) the same tuple that was originally missed.
Once that happens, relfrozenxid will completely catch up. (Actually, one
could reasonably argue that we never really "fell behind" in the first
place -- the amount of freezing needed to significantly advance
relfrozenxid won't have measurably increased at any point.  A once-off
drop in the extent to which VACUUM can advance relfrozenxid is almost
certainly harmless noise.)

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-WzkymFbz6D_vL+jmqSn_5q1wsFvFrE+37yLgL_Rkfd6Gzg@mail.gmail.com
---
 src/backend/access/heap/vacuumlazy.c | 84 ++++++++++++++++++++++++----
 1 file changed, 72 insertions(+), 12 deletions(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d481a300b..ea4b75189 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -169,6 +169,7 @@ typedef struct LVRelState
 
 	/* VACUUM operation's cutoff for pruning */
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
@@ -200,6 +201,7 @@ typedef struct LVRelState
 	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
 	BlockNumber frozenskipped_pages;	/* # frozen pages skipped via VM */
 	BlockNumber removed_pages;	/* # pages removed by relation truncation */
+	BlockNumber newly_frozen_pages; /* # pages with tuples frozen by us */
 	BlockNumber lpdead_item_pages;	/* # pages with LP_DEAD items */
 	BlockNumber missed_dead_pages;	/* # pages with missed dead tuples */
 	BlockNumber nonempty_pages; /* actually, last nonempty page + 1 */
@@ -474,6 +476,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 
 	/* Set cutoffs for entire VACUUM */
 	vacrel->OldestXmin = OldestXmin;
+	vacrel->OldestMxact = OldestMxact;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
 
@@ -654,12 +657,15 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 vacrel->relnamespace,
 							 vacrel->relname,
 							 vacrel->num_index_scans);
-			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u scanned (%.2f%% of total)\n"),
+			appendStringInfo(&buf, _("pages: %u removed, %u remain, %u scanned (%.2f%% of total), %u newly frozen (%.2f%% of total)\n"),
 							 vacrel->removed_pages,
 							 vacrel->rel_pages,
 							 vacrel->scanned_pages,
 							 orig_rel_pages == 0 ? 0 :
-							 100.0 * vacrel->scanned_pages / orig_rel_pages);
+							 100.0 * vacrel->scanned_pages / orig_rel_pages,
+							 vacrel->newly_frozen_pages,
+							 orig_rel_pages == 0 ? 0 :
+							 100.0 * vacrel->newly_frozen_pages / orig_rel_pages);
 			appendStringInfo(&buf,
 							 _("tuples: %lld removed, %lld remain, %lld are dead but not yet removable\n"),
 							 (long long) vacrel->tuples_deleted,
@@ -827,6 +833,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 	vacrel->scanned_pages = 0;
 	vacrel->frozenskipped_pages = 0;
 	vacrel->removed_pages = 0;
+	vacrel->newly_frozen_pages = 0;
 	vacrel->lpdead_item_pages = 0;
 	vacrel->missed_dead_pages = 0;
 	vacrel->nonempty_pages = 0;
@@ -1027,7 +1034,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 			/*
 			 * SKIP_PAGES_THRESHOLD (threshold for skipping) was not
 			 * crossed, or this is the last page.  Scan the page, even
-			 * though it's all-visible (and possibly even all-frozen).
+			 * though it's all-visible (and likely all-frozen, too).
 			 */
 			all_visible_according_to_vm = true;
 		}
@@ -1589,7 +1596,7 @@ lazy_scan_prune(LVRelState *vacrel,
 	ItemId		itemid;
 	HeapTupleData tuple;
 	HTSV_Result res;
-	int			tuples_deleted,
+	int			tuples_deleted = 0,
 				lpdead_items,
 				recently_dead_tuples,
 				num_tuples,
@@ -1600,6 +1607,9 @@ lazy_scan_prune(LVRelState *vacrel,
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
 	TransactionId NewRelfrozenxid;
 	MultiXactId NewRelminmxid;
+	TransactionId FreezeLimit = vacrel->FreezeLimit;
+	MultiXactId MultiXactCutoff = vacrel->MultiXactCutoff;
+	bool		freezeblk = false;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1610,7 +1620,6 @@ retry:
 	/* Initialize (or reset) page-level counters */
 	NewRelfrozenxid = vacrel->NewRelfrozenxid;
 	NewRelminmxid = vacrel->NewRelminmxid;
-	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
 	num_tuples = 0;
@@ -1625,9 +1634,9 @@ retry:
 	 * lpdead_items's final value can be thought of as the number of tuples
 	 * that were deleted from indexes.
 	 */
-	tuples_deleted = heap_page_prune(rel, buf, vistest,
-									 InvalidTransactionId, 0, &nnewlpdead,
-									 &vacrel->offnum);
+	tuples_deleted += heap_page_prune(rel, buf, vistest,
+									  InvalidTransactionId, 0, &nnewlpdead,
+									  &vacrel->offnum);
 
 	/*
 	 * Now scan the page to collect LP_DEAD items and check for tuples
@@ -1678,11 +1687,16 @@ retry:
 		 * vacrel->nonempty_pages value) is inherently race-prone.  It must be
 		 * treated as advisory/unreliable, so we might as well be slightly
 		 * optimistic.
+		 *
+		 * We delay setting all_visible to false due to seeing an LP_DEAD
+		 * item.  We need to test "is the page all_visible if we just consider
+		 * remaining tuples with tuple storage?" below, when considering if we
+		 * should freeze the tuples on the page.  (all_visible will be set to
+		 * false for caller once we've decided on what to freeze.)
 		 */
 		if (ItemIdIsDead(itemid))
 		{
 			deadoffsets[lpdead_items++] = offnum;
-			prunestate->all_visible = false;
 			prunestate->has_lpdead_items = true;
 			continue;
 		}
@@ -1816,8 +1830,8 @@ retry:
 		if (heap_prepare_freeze_tuple(tuple.t_data,
 									  vacrel->relfrozenxid,
 									  vacrel->relminmxid,
-									  vacrel->FreezeLimit,
-									  vacrel->MultiXactCutoff,
+									  FreezeLimit,
+									  MultiXactCutoff,
 									  &frozen[nfrozen],
 									  &tuple_totally_frozen,
 									  &NewRelfrozenxid,
@@ -1837,6 +1851,50 @@ retry:
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * Freeze the whole page using OldestXmin (not FreezeLimit) as our cutoff
+	 * if the page is now eligible to be marked all_visible (barring any
+	 * LP_DEAD items) when the page is not already eligible to be marked
+	 * all_frozen.  We generally expect to freeze all of a block's tuples
+	 * together and at once, or none at all.  FreezeLimit is just a backstop
+	 * mechanism that makes sure that we don't overlook one or two older
+	 * tuples.
+	 *
+	 * For example, it's just about possible that successive VACUUM operations
+	 * will never quite manage to use the main block-level logic to freeze one
+	 * old tuple from a page where all other tuples are continually updated.
+	 * We should not be in any hurry to freeze such a tuple.  Even still, it's
+	 * better if we take care of it before an anti-wraparound VACUUM becomes
+	 * necessary -- that would mean that we'd have to wait for a cleanup lock
+	 * during the aggressive VACUUM, which has risks of its own.
+	 *
+	 * FIXME This code structure has been used for prototyping and testing the
+	 * algorithm, details of which have settled.  Code itself to be rewritten,
+	 * though.  It is backwards right now -- should be _starting_ with
+	 * OldestXmin (not FreezeLimit), since that's what happens at the
+	 * conceptual level.
+	 *
+	 * TODO Make vacuum_freeze_min_age GUC/reloption default -1, which will be
+	 * interpreted as "whatever autovacuum_freeze_max_age/2 is".  Idea is to
+	 * make FreezeLimit into a true backstop, and to do our best to avoid
+	 * waiting for a cleanup lock (always prefer to punt to the next VACUUM,
+	 * since we can advance relfrozenxid to the oldest XID on the page inside
+	 * lazy_scan_noprune).
+	 */
+	if (!freezeblk &&
+		((nfrozen > 0 && nfrozen < num_tuples) ||
+		 (prunestate->all_visible && !prunestate->all_frozen)))
+	{
+		freezeblk = true;
+		FreezeLimit = vacrel->OldestXmin;
+		MultiXactCutoff = vacrel->OldestMxact;
+		goto retry;
+	}
+
+	/* Time to define all_visible in a way that accounts for LP_DEAD items */
+	if (lpdead_items > 0)
+		prunestate->all_visible = false;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
@@ -1854,6 +1912,8 @@ retry:
 	{
 		Assert(prunestate->hastup);
 
+		vacrel->newly_frozen_pages++;
+
 		/*
 		 * At least one tuple with storage needs to be frozen -- execute that
 		 * now.
@@ -1882,7 +1942,7 @@ retry:
 		{
 			XLogRecPtr	recptr;
 
-			recptr = log_heap_freeze(vacrel->rel, buf, vacrel->FreezeLimit,
+			recptr = log_heap_freeze(vacrel->rel, buf, FreezeLimit,
 									 frozen, nfrozen);
 			PageSetLSN(page, recptr);
 		}
-- 
2.30.2

From 6c8cb32e074e7de2414b067fcf4011acb4cca121 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Mon, 22 Nov 2021 10:02:30 -0800
Subject: [PATCH v8 1/3] Loosen coupling between relfrozenxid and tuple
 freezing.

The pg_class.relfrozenxid invariant for heap relations is as follows:
relfrozenxid must be less than or equal to the oldest extant XID in the
table, and must never wraparound (it must be advanced by VACUUM before
wraparound, or in extreme cases the system must be forced to stop
allocating new XIDs).

Before now, VACUUM always set relfrozenxid to whatever value it happened
to use when determining which tuples to freeze (the VACUUM operation's
FreezeLimit cutoff).  But there was no inherent reason why the oldest
extant XID in the table should be anywhere near as old as that.
Furthermore, even if it really was almost as old as FreezeLimit, that
tells us much more about the mechanism that VACUUM used to determine
which tuples to freeze than anything else.  Depending on the details of
the table and workload, it may have been possible to safely advance
relfrozenxid by many more XIDs, at a relatively small cost in freezing
(possibly no extra cost at all) -- but VACUUM rigidly coupled freezing
with advancing relfrozenxid, missing all this.

Teach VACUUM to track the newest possible safe final relfrozenxid
dynamically (and to track a new value for relminmxid).  In the extreme
though common case where all tuples are already frozen, or became frozen
(or were removed by pruning), the final relfrozenxid value will be
exactly equal to the OldestXmin value used by the same VACUUM operation.
A later patch will overhaul the strategy that VACUUM uses for freezing
so that relfrozenxid will tend to get set to a value that's relatively
close to OldestXmin in almost all cases.

Final relfrozenxid values still follow the same rules as before.  They
must still be >= FreezeLimit in an aggressive VACUUM.  Non-aggressive
VACUUMs can set relfrozenxid to any value that's greater than the
preexisting relfrozenxid, which could be either much earlier or much
later than FreezeLimit.  Much depends on workload characteristics.  In
practice there is significant natural variation that we can take
advantage of.

Credit for the general idea of using the oldest extant XID to set
pg_class.relfrozenxid at the end of VACUUM goes to Andres Freund.

Author: Peter Geoghegan <pg@bowt.ie>
Discussion: https://postgr.es/m/CAH2-WzkymFbz6D_vL+jmqSn_5q1wsFvFrE+37yLgL_Rkfd6Gzg@mail.gmail.com
---
 src/include/access/heapam.h          |   4 +-
 src/include/access/heapam_xlog.h     |   4 +-
 src/include/commands/vacuum.h        |   1 +
 src/backend/access/heap/heapam.c     | 186 ++++++++++++++++++++-------
 src/backend/access/heap/vacuumlazy.c |  85 ++++++++----
 src/backend/commands/cluster.c       |   5 +-
 src/backend/commands/vacuum.c        |  34 +++--
 7 files changed, 238 insertions(+), 81 deletions(-)

diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 0ad87730e..d35402f9f 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -168,7 +168,9 @@ extern bool heap_freeze_tuple(HeapTupleHeader tuple,
 							  TransactionId relfrozenxid, TransactionId relminmxid,
 							  TransactionId cutoff_xid, TransactionId cutoff_multi);
 extern bool heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-									MultiXactId cutoff_multi, Buffer buf);
+									MultiXactId cutoff_multi,
+									TransactionId *NewRelfrozenxid,
+									MultiXactId *NewRelminmxid, Buffer buf);
 extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple);
 
 extern void simple_heap_insert(Relation relation, HeapTuple tup);
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 5c47fdcec..ae55c90f7 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -410,7 +410,9 @@ extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 									  TransactionId cutoff_xid,
 									  TransactionId cutoff_multi,
 									  xl_heap_freeze_tuple *frz,
-									  bool *totally_frozen);
+									  bool *totally_frozen,
+									  TransactionId *NewRelfrozenxid,
+									  MultiXactId *NewRelminmxid);
 extern void heap_execute_freeze_tuple(HeapTupleHeader tuple,
 									  xl_heap_freeze_tuple *xlrec_tp);
 extern XLogRecPtr log_heap_visible(RelFileNode rnode, Buffer heap_buffer,
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index d64f6268f..ead88edda 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -291,6 +291,7 @@ extern bool vacuum_set_xid_limits(Relation rel,
 								  int multixact_freeze_min_age,
 								  int multixact_freeze_table_age,
 								  TransactionId *oldestXmin,
+								  MultiXactId *oldestMxact,
 								  TransactionId *freezeLimit,
 								  MultiXactId *multiXactCutoff);
 extern bool vacuum_xid_failsafe_check(TransactionId relfrozenxid,
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 98230aac4..d85a817ff 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -6087,12 +6087,24 @@ heap_inplace_update(Relation relation, HeapTuple tuple)
  * FRM_RETURN_IS_MULTI
  *		The return value is a new MultiXactId to set as new Xmax.
  *		(caller must obtain proper infomask bits using GetMultiXactIdHintBits)
+ *
+ * "NewRelfrozenxid" is an output value; it's used to maintain target new
+ * relfrozenxid for the relation.  It can be ignored unless "flags" contains
+ * either FRM_NOOP or FRM_RETURN_IS_MULTI, because we only handle multiXacts
+ * here.  This follows the general convention: only track XIDs that will still
+ * be in the table after the ongoing VACUUM finishes.  Note that it's up to
+ * caller to maintain this when the Xid return value is itself an Xid.
+ *
+ * Note that we cannot depend on xmin to maintain NewRelfrozenxid.  We need to
+ * push maintenance of NewRelfrozenxid down this far, since in general xmin
+ * might have been frozen by an earlier VACUUM operation, in which case our
+ * caller will not have factored-in xmin when maintaining NewRelfrozenxid.
  */
 static TransactionId
 FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 				  TransactionId relfrozenxid, TransactionId relminmxid,
 				  TransactionId cutoff_xid, MultiXactId cutoff_multi,
-				  uint16 *flags)
+				  uint16 *flags, TransactionId *NewRelfrozenxid)
 {
 	TransactionId xid = InvalidTransactionId;
 	int			i;
@@ -6104,6 +6116,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	bool		has_lockers;
 	TransactionId update_xid;
 	bool		update_committed;
+	TransactionId tempNewRelfrozenxid;
 
 	*flags = 0;
 
@@ -6198,13 +6211,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 
 	/* is there anything older than the cutoff? */
 	need_replace = false;
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	for (i = 0; i < nmembers; i++)
 	{
 		if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-		{
 			need_replace = true;
-			break;
-		}
+		if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+			tempNewRelfrozenxid = members[i].xid;
 	}
 
 	/*
@@ -6213,6 +6226,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 */
 	if (!need_replace)
 	{
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 		*flags |= FRM_NOOP;
 		pfree(members);
 		return InvalidTransactionId;
@@ -6222,6 +6236,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 	 * If the multi needs to be updated, figure out which members do we need
 	 * to keep.
 	 */
+	tempNewRelfrozenxid = *NewRelfrozenxid;
 	nnewmembers = 0;
 	newmembers = palloc(sizeof(MultiXactMember) * nmembers);
 	has_lockers = false;
@@ -6303,7 +6318,11 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			 * list.)
 			 */
 			if (TransactionIdIsValid(update_xid))
+			{
 				newmembers[nnewmembers++] = members[i];
+				if (TransactionIdPrecedes(members[i].xid, tempNewRelfrozenxid))
+					tempNewRelfrozenxid = members[i].xid;
+			}
 		}
 		else
 		{
@@ -6313,6 +6332,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 			{
 				/* running locker cannot possibly be older than the cutoff */
 				Assert(!TransactionIdPrecedes(members[i].xid, cutoff_xid));
+				Assert(!TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid));
 				newmembers[nnewmembers++] = members[i];
 				has_lockers = true;
 			}
@@ -6341,6 +6361,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		if (update_committed)
 			*flags |= FRM_MARK_COMMITTED;
 		xid = update_xid;
+		/* Caller manages NewRelfrozenxid directly when we return an XID */
 	}
 	else
 	{
@@ -6350,6 +6371,7 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
 		 */
 		xid = MultiXactIdCreateFromMembers(nnewmembers, newmembers);
 		*flags |= FRM_RETURN_IS_MULTI;
+		*NewRelfrozenxid = tempNewRelfrozenxid;
 	}
 
 	pfree(newmembers);
@@ -6368,6 +6390,13 @@ FreezeMultiXactId(MultiXactId multi, uint16 t_infomask,
  * will be totally frozen after these operations are performed and false if
  * more freezing will eventually be required.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will actually go on to freeze as indicated by our *frz output, so
+ * any (xmin, xmax, xvac) XIDs that we indicate need to be frozen won't need
+ * to be counted here.  Values are valid lower bounds at the point that the
+ * ongoing VACUUM finishes.
+ *
  * Caller is responsible for setting the offset field, if appropriate.
  *
  * It is assumed that the caller has checked the tuple with
@@ -6392,7 +6421,9 @@ bool
 heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 						  TransactionId relfrozenxid, TransactionId relminmxid,
 						  TransactionId cutoff_xid, TransactionId cutoff_multi,
-						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p)
+						  xl_heap_freeze_tuple *frz, bool *totally_frozen_p,
+						  TransactionId *NewRelfrozenxid,
+						  MultiXactId *NewRelminmxid)
 {
 	bool		changed = false;
 	bool		xmax_already_frozen = false;
@@ -6436,6 +6467,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->t_infomask |= HEAP_XMIN_FROZEN;
 			changed = true;
 		}
+		else if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+		{
+			/* won't be frozen, but older than current NewRelfrozenxid */
+			*NewRelfrozenxid = xid;
+		}
 	}
 
 	/*
@@ -6453,10 +6489,11 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 	{
 		TransactionId newxmax;
 		uint16		flags;
+		TransactionId temp = *NewRelfrozenxid;
 
 		newxmax = FreezeMultiXactId(xid, tuple->t_infomask,
 									relfrozenxid, relminmxid,
-									cutoff_xid, cutoff_multi, &flags);
+									cutoff_xid, cutoff_multi, &flags, &temp);
 
 		freeze_xmax = (flags & FRM_INVALIDATE_XMAX);
 
@@ -6474,6 +6511,24 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			if (flags & FRM_MARK_COMMITTED)
 				frz->t_infomask |= HEAP_XMAX_COMMITTED;
 			changed = true;
+
+			if (TransactionIdPrecedes(newxmax, *NewRelfrozenxid))
+			{
+				/* New xmax is an XID older than new NewRelfrozenxid */
+				*NewRelfrozenxid = newxmax;
+			}
+		}
+		else if (flags & FRM_NOOP)
+		{
+			/*
+			 * Changing nothing, so might have to ratchet back NewRelminmxid,
+			 * NewRelfrozenxid, or both together
+			 */
+			if (MultiXactIdIsValid(xid) &&
+				MultiXactIdPrecedes(xid, *NewRelminmxid))
+				*NewRelminmxid = xid;
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 		else if (flags & FRM_RETURN_IS_MULTI)
 		{
@@ -6495,6 +6550,13 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			frz->xmax = newxmax;
 
 			changed = true;
+
+			/*
+			 * New multixact might have remaining XID older than
+			 * NewRelfrozenxid
+			 */
+			if (TransactionIdPrecedes(temp, *NewRelfrozenxid))
+				*NewRelfrozenxid = temp;
 		}
 	}
 	else if (TransactionIdIsNormal(xid))
@@ -6522,7 +6584,14 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 			freeze_xmax = true;
 		}
 		else
+		{
 			freeze_xmax = false;
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			{
+				/* won't be frozen, but older than current NewRelfrozenxid */
+				*NewRelfrozenxid = xid;
+			}
+		}
 	}
 	else if ((tuple->t_infomask & HEAP_XMAX_INVALID) ||
 			 !TransactionIdIsValid(HeapTupleHeaderGetRawXmax(tuple)))
@@ -6569,6 +6638,9 @@ heap_prepare_freeze_tuple(HeapTupleHeader tuple,
 		 * was removed in PostgreSQL 9.0.  Note that if we were to respect
 		 * cutoff_xid here, we'd need to make surely to clear totally_frozen
 		 * when we skipped freezing on that basis.
+		 *
+		 * Since we always freeze here, NewRelfrozenxid doesn't need to be
+		 * maintained.
 		 */
 		if (TransactionIdIsNormal(xid))
 		{
@@ -6646,11 +6718,14 @@ heap_freeze_tuple(HeapTupleHeader tuple,
 	xl_heap_freeze_tuple frz;
 	bool		do_freeze;
 	bool		tuple_totally_frozen;
+	TransactionId NewRelfrozenxid = FirstNormalTransactionId;
+	MultiXactId NewRelminmxid = FirstMultiXactId;
 
 	do_freeze = heap_prepare_freeze_tuple(tuple,
 										  relfrozenxid, relminmxid,
 										  cutoff_xid, cutoff_multi,
-										  &frz, &tuple_totally_frozen);
+										  &frz, &tuple_totally_frozen,
+										  &NewRelfrozenxid, &NewRelminmxid);
 
 	/*
 	 * Note that because this is not a WAL-logged operation, we don't need to
@@ -7080,6 +7155,15 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  * Check to see whether any of the XID fields of a tuple (xmin, xmax, xvac)
  * are older than the specified cutoff XID or MultiXactId.  If so, return true.
  *
+ * Also maintains *NewRelfrozenxid and *NewRelminmxid, which are the current
+ * target relfrozenxid and relminmxid for the relation.  Assumption is that
+ * caller will never freeze any of the XIDs from the tuple, even when we say
+ * that they should.  If caller opts to go with our recommendation to freeze,
+ * then it must account for the fact that it shouldn't trust how we've set
+ * NewRelfrozenxid/NewRelminmxid.  (In practice aggressive VACUUMs always take
+ * our recommendation because they must, and non-aggressive VACUUMs always opt
+ * to not freeze, preferring to ratchet back NewRelfrozenxid instead).
+ *
  * It doesn't matter whether the tuple is alive or dead, we are checking
  * to see if a tuple needs to be removed or frozen to avoid wraparound.
  *
@@ -7088,74 +7172,86 @@ heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple)
  */
 bool
 heap_tuple_needs_freeze(HeapTupleHeader tuple, TransactionId cutoff_xid,
-						MultiXactId cutoff_multi, Buffer buf)
+						MultiXactId cutoff_multi,
+						TransactionId *NewRelfrozenxid,
+						MultiXactId *NewRelminmxid, Buffer buf)
 {
 	TransactionId xid;
+	bool		needs_freeze = false;
 
 	xid = HeapTupleHeaderGetXmin(tuple);
-	if (TransactionIdIsNormal(xid) &&
-		TransactionIdPrecedes(xid, cutoff_xid))
-		return true;
+	if (TransactionIdIsNormal(xid))
+	{
+		if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+			*NewRelfrozenxid = xid;
+		if (TransactionIdPrecedes(xid, cutoff_xid))
+			needs_freeze = true;
+	}
 
 	/*
 	 * The considerations for multixacts are complicated; look at
 	 * heap_prepare_freeze_tuple for justifications.  This routine had better
 	 * be in sync with that one!
+	 *
+	 * (Actually, we maintain NewRelminmxid differently here, because we
+	 * assume that XIDs that should be frozen according to cutoff_xid won't
+	 * be, whereas heap_prepare_freeze_tuple makes the opposite assumption.)
 	 */
 	if (tuple->t_infomask & HEAP_XMAX_IS_MULTI)
 	{
 		MultiXactId multi;
+		MultiXactMember *members;
+		int			nmembers;
 
 		multi = HeapTupleHeaderGetRawXmax(tuple);
-		if (!MultiXactIdIsValid(multi))
-		{
-			/* no xmax set, ignore */
-			;
-		}
-		else if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
+		if (MultiXactIdIsValid(multi) &&
+			MultiXactIdPrecedes(multi, *NewRelminmxid))
+			*NewRelminmxid = multi;
+
+		if (HEAP_LOCKED_UPGRADED(tuple->t_infomask))
 			return true;
 		else if (MultiXactIdPrecedes(multi, cutoff_multi))
-			return true;
-		else
+			needs_freeze = true;
+
+		/* need to check whether any member of the mxact is too old */
+		nmembers = GetMultiXactIdMembers(multi, &members, false,
+										 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
+
+		for (int i = 0; i < nmembers; i++)
 		{
-			MultiXactMember *members;
-			int			nmembers;
-			int			i;
-
-			/* need to check whether any member of the mxact is too old */
-
-			nmembers = GetMultiXactIdMembers(multi, &members, false,
-											 HEAP_XMAX_IS_LOCKED_ONLY(tuple->t_infomask));
-
-			for (i = 0; i < nmembers; i++)
-			{
-				if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
-				{
-					pfree(members);
-					return true;
-				}
-			}
-			if (nmembers > 0)
-				pfree(members);
+			if (TransactionIdPrecedes(members[i].xid, cutoff_xid))
+				needs_freeze = true;
+			if (TransactionIdPrecedes(members[i].xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
 		}
+		if (nmembers > 0)
+			pfree(members);
 	}
 	else
 	{
 		xid = HeapTupleHeaderGetRawXmax(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
 	if (tuple->t_infomask & HEAP_MOVED)
 	{
 		xid = HeapTupleHeaderGetXvac(tuple);
-		if (TransactionIdIsNormal(xid) &&
-			TransactionIdPrecedes(xid, cutoff_xid))
-			return true;
+		if (TransactionIdIsNormal(xid))
+		{
+			if (TransactionIdPrecedes(xid, *NewRelfrozenxid))
+				*NewRelfrozenxid = xid;
+			if (TransactionIdPrecedes(xid, cutoff_xid))
+				needs_freeze = true;
+		}
 	}
 
-	return false;
+	return needs_freeze;
 }
 
 /*
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d57055674..d481a300b 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -172,8 +172,10 @@ typedef struct LVRelState
 	/* VACUUM operation's cutoff for freezing XIDs and MultiXactIds */
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
-	/* Are FreezeLimit/MultiXactCutoff still valid? */
-	bool		freeze_cutoffs_valid;
+
+	/* Track new pg_class.relfrozenxid/pg_class.relminmxid values */
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	/* Error reporting state */
 	char	   *relnamespace;
@@ -330,6 +332,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	PgStat_Counter startreadtime = 0;
 	PgStat_Counter startwritetime = 0;
 	TransactionId OldestXmin;
+	MultiXactId OldestMxact;
 	TransactionId FreezeLimit;
 	MultiXactId MultiXactCutoff;
 
@@ -365,8 +368,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 									   params->freeze_table_age,
 									   params->multixact_freeze_min_age,
 									   params->multixact_freeze_table_age,
-									   &OldestXmin, &FreezeLimit,
-									   &MultiXactCutoff);
+									   &OldestXmin, &OldestMxact,
+									   &FreezeLimit, &MultiXactCutoff);
 
 	skipwithvm = true;
 	if (params->options & VACOPT_DISABLE_PAGE_SKIPPING)
@@ -473,8 +476,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	vacrel->OldestXmin = OldestXmin;
 	vacrel->FreezeLimit = FreezeLimit;
 	vacrel->MultiXactCutoff = MultiXactCutoff;
-	/* Track if cutoffs became invalid (possible in !aggressive case only) */
-	vacrel->freeze_cutoffs_valid = true;
+
+	/* Initialize values used to advance relfrozenxid/relminmxid at the end */
+	vacrel->NewRelfrozenxid = OldestXmin;
+	vacrel->NewRelminmxid = OldestMxact;
 
 	/*
 	 * Call lazy_scan_heap to perform all required heap pruning, index
@@ -527,16 +532,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * Aggressive VACUUM must reliably advance relfrozenxid (and relminmxid).
 	 * We are able to advance relfrozenxid in a non-aggressive VACUUM too,
 	 * provided we didn't skip any all-visible (not all-frozen) pages using
-	 * the visibility map, and assuming that we didn't fail to get a cleanup
-	 * lock that made it unsafe with respect to FreezeLimit (or perhaps our
-	 * MultiXactCutoff) established for VACUUM operation.
+	 * the visibility map.  A non-aggressive VACUUM might only be able to
+	 * advance relfrozenxid to an XID from before FreezeLimit (or a relminmxid
+	 * from before MultiXactCutoff) when it wasn't possible to freeze some
+	 * tuples due to our inability to acquire a cleanup lock, but the effect
+	 * is usually insignificant -- NewRelfrozenxid value still has a decent
+	 * chance of being much more recent that the existing relfrozenxid.
 	 *
 	 * NB: We must use orig_rel_pages, not vacrel->rel_pages, since we want
 	 * the rel_pages used by lazy_scan_heap, which won't match when we
 	 * happened to truncate the relation afterwards.
 	 */
-	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages ||
-		!vacrel->freeze_cutoffs_valid)
+	if (vacrel->scanned_pages + vacrel->frozenskipped_pages < orig_rel_pages)
 	{
 		/* Cannot advance relfrozenxid/relminmxid */
 		Assert(!aggressive);
@@ -548,11 +555,23 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	}
 	else
 	{
+		/*
+		 * Aggressive case is strictly required to advance relfrozenxid, at
+		 * least up to FreezeLimit (same applies with relminmxid and its
+		 * cutoff, MultiXactCutoff).  Assert that we got this right now.
+		 */
 		Assert(vacrel->scanned_pages + vacrel->frozenskipped_pages ==
 			   orig_rel_pages);
+		Assert(!aggressive ||
+			   TransactionIdPrecedesOrEquals(FreezeLimit,
+											 vacrel->NewRelfrozenxid));
+		Assert(!aggressive ||
+			   MultiXactIdPrecedesOrEquals(MultiXactCutoff,
+										   vacrel->NewRelminmxid));
+
 		vac_update_relstats(rel, new_rel_pages, new_live_tuples,
 							new_rel_allvisible, vacrel->nindexes > 0,
-							FreezeLimit, MultiXactCutoff,
+							vacrel->NewRelfrozenxid, vacrel->NewRelminmxid,
 							&frozenxid_updated, &minmulti_updated, false);
 	}
 
@@ -657,17 +676,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 							 OldestXmin, diff);
 			if (frozenxid_updated)
 			{
-				diff = (int32) (FreezeLimit - vacrel->relfrozenxid);
+				diff = (int32) (vacrel->NewRelfrozenxid - vacrel->relfrozenxid);
 				appendStringInfo(&buf,
 								 _("new relfrozenxid: %u, which is %d xids ahead of previous value\n"),
-								 FreezeLimit, diff);
+								 vacrel->NewRelfrozenxid, diff);
 			}
 			if (minmulti_updated)
 			{
-				diff = (int32) (MultiXactCutoff - vacrel->relminmxid);
+				diff = (int32) (vacrel->NewRelminmxid - vacrel->relminmxid);
 				appendStringInfo(&buf,
 								 _("new relminmxid: %u, which is %d mxids ahead of previous value\n"),
-								 MultiXactCutoff, diff);
+								 vacrel->NewRelminmxid, diff);
 			}
 			if (orig_rel_pages > 0)
 			{
@@ -1579,6 +1598,8 @@ lazy_scan_prune(LVRelState *vacrel,
 	int			nfrozen;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	xl_heap_freeze_tuple frozen[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid;
+	MultiXactId NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -1587,6 +1608,8 @@ lazy_scan_prune(LVRelState *vacrel,
 retry:
 
 	/* Initialize (or reset) page-level counters */
+	NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	NewRelminmxid = vacrel->NewRelminmxid;
 	tuples_deleted = 0;
 	lpdead_items = 0;
 	recently_dead_tuples = 0;
@@ -1796,7 +1819,9 @@ retry:
 									  vacrel->FreezeLimit,
 									  vacrel->MultiXactCutoff,
 									  &frozen[nfrozen],
-									  &tuple_totally_frozen))
+									  &tuple_totally_frozen,
+									  &NewRelfrozenxid,
+									  &NewRelminmxid))
 		{
 			/* Will execute freeze below */
 			frozen[nfrozen++].offset = offnum;
@@ -1810,13 +1835,16 @@ retry:
 			prunestate->all_frozen = false;
 	}
 
+	vacrel->offnum = InvalidOffsetNumber;
+
 	/*
 	 * We have now divided every item on the page into either an LP_DEAD item
 	 * that will need to be vacuumed in indexes later, or a LP_NORMAL tuple
 	 * that remains and needs to be considered for freezing now (LP_UNUSED and
 	 * LP_REDIRECT items also remain, but are of no further interest to us).
 	 */
-	vacrel->offnum = InvalidOffsetNumber;
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
 
 	/*
 	 * Consider the need to freeze any items with tuple storage from the page
@@ -1969,6 +1997,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 				missed_dead_tuples;
 	HeapTupleHeader tupleheader;
 	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+	TransactionId NewRelfrozenxid = vacrel->NewRelfrozenxid;
+	MultiXactId NewRelminmxid = vacrel->NewRelminmxid;
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
 
@@ -2015,7 +2045,8 @@ lazy_scan_noprune(LVRelState *vacrel,
 		tupleheader = (HeapTupleHeader) PageGetItem(page, itemid);
 		if (heap_tuple_needs_freeze(tupleheader,
 									vacrel->FreezeLimit,
-									vacrel->MultiXactCutoff, buf))
+									vacrel->MultiXactCutoff,
+									&NewRelfrozenxid, &NewRelminmxid, buf))
 		{
 			if (vacrel->aggressive)
 			{
@@ -2025,10 +2056,12 @@ lazy_scan_noprune(LVRelState *vacrel,
 			}
 
 			/*
-			 * Current non-aggressive VACUUM operation definitely won't be
-			 * able to advance relfrozenxid or relminmxid
+			 * A non-aggressive VACUUM doesn't have to wait on a cleanup lock
+			 * to ensure that it advances relfrozenxid to a sufficiently
+			 * recent XID that happens to be present on this page.  It can
+			 * just accept an older New/final relfrozenxid instead.  There is
+			 * a decent chance that the problem will go away naturally.
 			 */
-			vacrel->freeze_cutoffs_valid = false;
 		}
 
 		num_tuples++;
@@ -2078,6 +2111,14 @@ lazy_scan_noprune(LVRelState *vacrel,
 
 	vacrel->offnum = InvalidOffsetNumber;
 
+	/*
+	 * We have committed to not freezing the tuples on this page (always
+	 * happens with a non-aggressive VACUUM), so make sure that the target
+	 * relfrozenxid/relminmxid values reflect the XIDs/MXIDs we encountered
+	 */
+	vacrel->NewRelfrozenxid = NewRelfrozenxid;
+	vacrel->NewRelminmxid = NewRelminmxid;
+
 	/*
 	 * Now save details of the LP_DEAD items from the page in vacrel (though
 	 * only when VACUUM uses two-pass strategy)
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 02a7e94bf..a7e988298 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -767,6 +767,7 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	TupleDesc	oldTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TupleDesc	newTupDesc PG_USED_FOR_ASSERTS_ONLY;
 	TransactionId OldestXmin;
+	MultiXactId oldestMxact;
 	TransactionId FreezeXid;
 	MultiXactId MultiXactCutoff;
 	bool		use_sort;
@@ -856,8 +857,8 @@ copy_table_data(Oid OIDNewHeap, Oid OIDOldHeap, Oid OIDOldIndex, bool verbose,
 	 * Since we're going to rewrite the whole table anyway, there's no reason
 	 * not to be aggressive about this.
 	 */
-	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0,
-						  &OldestXmin, &FreezeXid, &MultiXactCutoff);
+	vacuum_set_xid_limits(OldHeap, 0, 0, 0, 0, &OldestXmin, &oldestMxact,
+						  &FreezeXid, &MultiXactCutoff);
 
 	/*
 	 * FreezeXid will become the table's new relfrozenxid, and that mustn't go
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index b6767a5ff..d71ff21b1 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -945,14 +945,26 @@ get_all_vacuum_rels(int options)
  * The output parameters are:
  * - oldestXmin is the Xid below which tuples deleted by any xact (that
  *   committed) should be considered DEAD, not just RECENTLY_DEAD.
- * - freezeLimit is the Xid below which all Xids are replaced by
- *	 FrozenTransactionId during vacuum.
- * - multiXactCutoff is the value below which all MultiXactIds are removed
- *   from Xmax.
+ * - oldestMxact is the Mxid below which MultiXacts are definitely not
+ *   seen as visible by any running transaction.
+ * - freezeLimit is the Xid below which all Xids are definitely replaced by
+ *   FrozenTransactionId during aggressive vacuums.
+ * - multiXactCutoff is the value below which all MultiXactIds are definitely
+ *   removed from Xmax during aggressive vacuums.
  *
  * Return value indicates if vacuumlazy.c caller should make its VACUUM
  * operation aggressive.  An aggressive VACUUM must advance relfrozenxid up to
- * FreezeLimit, and relminmxid up to multiXactCutoff.
+ * FreezeLimit (at a minimum), and relminmxid up to multiXactCutoff (at a
+ * minimum).
+ *
+ * oldestXmin and oldestMxact can be thought of as the most recent values that
+ * can ever be passed to vac_update_relstats() as frozenxid and minmulti
+ * arguments.  These exact values can be used when no newer XIDs or MultiXacts
+ * remain in the heap relation (e.g., with an empty table).  It's typical for
+ * vacuumlazy.c caller to notice that older XIDs/Multixacts remain in the
+ * table, which will force it to use the oldest extant values when it calls
+ * vac_update_relstats().  Ideally these values won't be very far behind the
+ * "optimal" oldestXmin and oldestMxact values we provide.
  */
 bool
 vacuum_set_xid_limits(Relation rel,
@@ -961,6 +973,7 @@ vacuum_set_xid_limits(Relation rel,
 					  int multixact_freeze_min_age,
 					  int multixact_freeze_table_age,
 					  TransactionId *oldestXmin,
+					  MultiXactId *oldestMxact,
 					  TransactionId *freezeLimit,
 					  MultiXactId *multiXactCutoff)
 {
@@ -969,7 +982,6 @@ vacuum_set_xid_limits(Relation rel,
 	int			effective_multixact_freeze_max_age;
 	TransactionId limit;
 	TransactionId safeLimit;
-	MultiXactId oldestMxact;
 	MultiXactId mxactLimit;
 	MultiXactId safeMxactLimit;
 	int			freezetable;
@@ -1065,9 +1077,11 @@ vacuum_set_xid_limits(Relation rel,
 						 effective_multixact_freeze_max_age / 2);
 	Assert(mxid_freezemin >= 0);
 
+	/* Remember for caller */
+	*oldestMxact = GetOldestMultiXactId();
+
 	/* compute the cutoff multi, being careful to generate a valid value */
-	oldestMxact = GetOldestMultiXactId();
-	mxactLimit = oldestMxact - mxid_freezemin;
+	mxactLimit = *oldestMxact - mxid_freezemin;
 	if (mxactLimit < FirstMultiXactId)
 		mxactLimit = FirstMultiXactId;
 
@@ -1082,8 +1096,8 @@ vacuum_set_xid_limits(Relation rel,
 				(errmsg("oldest multixact is far in the past"),
 				 errhint("Close open transactions with multixacts soon to avoid wraparound problems.")));
 		/* Use the safe limit, unless an older mxact is still running */
-		if (MultiXactIdPrecedes(oldestMxact, safeMxactLimit))
-			mxactLimit = oldestMxact;
+		if (MultiXactIdPrecedes(*oldestMxact, safeMxactLimit))
+			mxactLimit = *oldestMxact;
 		else
 			mxactLimit = safeMxactLimit;
 	}
-- 
2.30.2

From 3f01281af3ba81b35777cb7d717f76e001fd3e10 Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 19 Feb 2022 14:07:35 -0800
Subject: [PATCH] Add adversarial ConditionalLockBufferForCleanup() gizmo to
 vacuumlazy.c.

---
 src/backend/access/heap/vacuumlazy.c | 36 +++++++++++++++++++++++++++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 242511a23..31c6b360e 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -50,6 +50,7 @@
 #include "commands/dbcommands.h"
 #include "commands/progress.h"
 #include "commands/vacuum.h"
+#include "common/pg_prng.h"
 #include "executor/instrument.h"
 #include "miscadmin.h"
 #include "optimizer/paths.h"
@@ -748,6 +749,39 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	}
 }
 
+/*
+ * Adversarial gizmo, simulates excessive failure to get cleanup locks
+ */
+static inline bool
+lazy_conditionallockbufferforcleanup(Buffer buffer)
+{
+	/*
+	 * Artificially fail to get a cleanup lock 50% of the time.
+	 *
+	 * XXX: What about temp tables?  We simulate not getting a cleanup lock
+	 * there, but is that choice actually reasonable?
+	 */
+	if (pg_prng_uint32(&pg_global_prng_state) <= (PG_UINT32_MAX / 2))
+		return false;
+
+#if 0
+	/*
+	 * 50% is very very aggressive, while 2% - 5% is still basically
+	 * adversarial but in many ways less annoying.
+	 *
+	 * This version (which injects a failure to get a cleanup lock 2% of the
+	 * time) seems to pass the regression tests, even with my parallel make
+	 * check-world recipe.  Expected query plans don't seem to shift on
+	 * account of unexpected index bloat (nor are there any problems of a
+	 * similar nature) with this variant of the gizmo.
+	 */
+	if (pg_prng_uint32(&pg_global_prng_state) <= (PG_UINT32_MAX / 50))
+		return false;
+#endif
+
+	return ConditionalLockBufferForCleanup(buffer);
+}
+
 /*
  *	lazy_scan_heap() -- workhorse function for VACUUM
  *
@@ -1093,7 +1127,7 @@ lazy_scan_heap(LVRelState *vacrel, int nworkers)
 		 * a cleanup lock right away, we may be able to settle for reduced
 		 * processing using lazy_scan_noprune.
 		 */
-		if (!ConditionalLockBufferForCleanup(buf))
+		if (!lazy_conditionallockbufferforcleanup(buf))
 		{
 			bool		hastup,
 						recordfreespace;
-- 
2.30.2

From 81f01ca623b115647ee78a1b09bbb4458fb35dab Mon Sep 17 00:00:00 2001
From: Peter Geoghegan <pg@bowt.ie>
Date: Sat, 19 Feb 2022 16:13:48 -0800
Subject: [PATCH 2/2] Fix for pg_surgery's heap_force_kill() function.

---
 contrib/pg_surgery/heap_surgery.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/contrib/pg_surgery/heap_surgery.c b/contrib/pg_surgery/heap_surgery.c
index 3e641aa64..a3a193ba5 100644
--- a/contrib/pg_surgery/heap_surgery.c
+++ b/contrib/pg_surgery/heap_surgery.c
@@ -311,7 +311,8 @@ heap_force_common(FunctionCallInfo fcinfo, HeapTupleForceOption heap_force_opt)
 		 */
 		if (did_modify_page)
 		{
-			/* Mark buffer dirty before we write WAL. */
+			/* Defragment and mark buffer dirty before we write WAL. */
+			PageRepairFragmentation(page);
 			MarkBufferDirty(buf);
 
 			/* XLOG stuff */
-- 
2.30.2

���bcorrupt-hot-chain.page��1@@F�w���B�J�q��	GQ:
�C,����K�x�)F���KUyuGpZ�^�5�������Z����-������������^���	[b�j