Trigger more frequent autovacuums of heavy insert tables

Started by Melanie Plagemanabout 1 year ago50 messages
#1Melanie Plageman
melanieplageman@gmail.com
2 attachment(s)

Hi,

Because of the way autovacuum_vacuum_[insert]_scale_factor works,
autovacuums trigger less frequently as the relation gets larger.

See this math in relation_needs_vacanalyze:

vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;

For an insert-only table, nearly all the unvacuumed pages will be
eligible to be set all-visible and many will be eligible to be set
all-frozen.

Because normal vacuums can skip all-visible pages, proactively setting
these pages all-visible by vacuuming them sooner often reduces IO
overhead, as they are more likely to still be in shared buffers the
sooner they are vacuumed after last being touched.

Vacuuming these pages more proactively and setting them frozen also
helps amortize the work of aggressive vacuums -- which often
negatively impact the performance of the system.

By considering only the unfrozen portion of the table when calculating
the vacuum insert threshold, we can trigger vacuums more proactively
on insert-heavy tables. This changes the definition of
insert_scale_factor to a percentage of "active" table size. The
attached patch does this.

I've estimated the unfrozen percentage of the table by adding a new
field to pg_class, relallfrozen, which is updated in the same places
as relallvisible.

As an example of this patch in action, I designed a benchmark in which
a table is bulk-loaded with 1 GB of data with COPY FREEZE. Then I run
a custom pgbench with four clients inserting 10 tuples per transaction
into the table for 1_000_000 transactions each.

Note that I configured Postgres to try and observe the effects of this
patch on a compressed timeline. At the bottom of the mail, I've
included details on all of the GUCs I set and why.

Over the course of the same number of transactions, master triggered 8
autovacuums of the table and the patch triggered 16.

With the patch, despite doing twice as many vacuums, autovacuum
workers did 10% fewer reads and 93% fewer writes.

At the end of the benchmark, the patched version of Postgres had
emitted twice as many FPIs as master.

More frequent vacuums means each vacuum scans fewer pages, but, more
interestingly, the first vacuum after a checkpoint is much more
efficient. With the patch, the first vacuum after a checkpoint emits
half as many FPIs. You can see that only 18 pages were newly dirtied.
So, with the patch, the pages being vacuumed are usually still in
shared buffers and still dirty.

Master
------
2024-10-22 13:53:14.293 EDT [3594] LOG: checkpoint starting: time
2024-10-22 13:53:27.849 EDT [3964] LOG: automatic vacuum of table "history"
pages: 0 removed, 753901 remain, 151589 scanned (20.11% of total)
I/O timings: read: 77.962 ms, write: 92.879 ms
avg read rate: 95.840 MB/s, avg write rate: 96.852 MB/s
buffer usage: 268318 hits, 35133 reads, 35504 dirtied
WAL usage: 218320 records, 98672 full page images, 71314906 bytes

Patch
-----
2024-10-22 13:48:43.951 EDT [1471] LOG: checkpoint starting: time
2024-10-22 13:48:59.741 EDT [1802] LOG: automatic vacuum of table "history"
pages: 0 removed, 774375 remain, 121367 scanned (15.67% of total)
I/O timings: read: 2.974 ms, write: 4.434 ms
avg read rate: 1.363 MB/s, avg write rate: 0.126 MB/s
buffer usage: 242817 hits, 195 reads, 18 dirtied
WAL usage: 121389 records, 49216 full page images, 34408291 bytes

While it is true that timing will change significantly from run to
run, I observed over many runs that the more frequent vacuums of the
table led to less overall overhead due to vacuuming pages before they
are evicted from shared buffers.

Below is a detailed description of the benchmark and Postgres configuration:

Benchmark
=========

Set these GUCs:

-- initial table data should fill shared buffers
shared_buffers=1GB
-- give us a chance to try and vacuum the table a bunch of times
autovacuum_naptime=2

-- all checkpoints should be triggered by timing
max/min_wal_size=150GB
-- let's get at least 1 checkpoint during the short benchmark
checkpoint_timeout='2min'

-- let's not be bottlenecked on WAL I/O
wal_buffers='128MB'
wal_compression='zstd'

-- let's get a lot of inserts done quickly
synchronous_commit='off'

-- let's not take too many breaks for vacuum delay
vacuum_cost_limit = 2000

-- so we can see what happened
log_checkpoints = on
log_autovacuum_min_duration=0

-- so we can get more stats
track_wal_io_timing=on
track_io_timing = on

First I created the table that you will see later in DDL and loaded it
by running pgbench in the same way as I do in the benchmark until
there was 1 GB of table data. Then I copied that out to a file
'history.data'

I included an index because the more up-to-date visibility map would
benefit index-only scans -- which you could add to the benchmark if
you want.

DDL
--
BEGIN;
DROP TABLE IF EXISTS history;
CREATE TABLE history(
id BIGINT,
client_id INT NOT NULL,
mtime TIMESTAMPTZ DEFAULT NOW(),
data TEXT);

COPY history FROM 'history.data' WITH (freeze on);
CREATE INDEX ON history(id);
COMMIT;

pgbench \
--random-seed=0 \
--no-vacuum \
-M prepared \
-c 4 \
-j 4 \
-t 1000000 \
-R 27000 \
-f- <<EOF
INSERT INTO history(id, client_id, data)
VALUES
(:client_id, :client_id, repeat('a', 90)),
(:client_id, :client_id, repeat('b', 90)),
(:client_id, :client_id, repeat('c', 90)),
(:client_id, :client_id, repeat('d', 90)),
(:client_id, :client_id, repeat('e', 90)),
(:client_id, :client_id, repeat('f', 90)),
(:client_id, :client_id, repeat('g', 90)),
(:client_id, :client_id, repeat('h', 90)),
(:client_id, :client_id, repeat('i', 90)),
(:client_id, :client_id, repeat('j', 90));
EOF

- Melanie

Attachments:

v1-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchapplication/octet-stream; name=v1-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From 6de3d9c7c1b1203a1b7680b74bef70fa7ab6b653 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 4 Oct 2024 17:06:04 -0400
Subject: [PATCH v1 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.
---
 doc/src/sgml/config.sgml                      |  4 +--
 src/backend/postmaster/autovacuum.c           | 26 +++++++++++++++++--
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 934ef5e469..a303f06e4c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8683,10 +8683,10 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
       </term>
       <listitem>
        <para>
-        Specifies a fraction of the table size to add to
+        Specifies a fraction of the active (unfrozen) table size to add to
         <varname>autovacuum_vacuum_insert_threshold</varname>
         when deciding whether to trigger a <command>VACUUM</command>.
-        The default is 0.2 (20% of table size).
+        The default is 0.2 (20% of active table size).
         This parameter can only be set in the <filename>postgresql.conf</filename>
         file or on the server command line;
         but the setting can be overridden for individual tables by
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index dc3cf87aba..9bd96794ae 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2922,7 +2922,12 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
+
+	/* From pg_class */
+	float4		reltuples;
+	int32		relpages;
+	int32		relallfrozen;
+	int32		relallvisible;
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3030,6 +3035,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
+		float4		pcnt_unfrozen = 1;
+
+		relpages = classForm->relpages;
+		relallfrozen = classForm->relallfrozen;
+		relallvisible = classForm->relallvisible;
 		reltuples = classForm->reltuples;
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
@@ -3039,8 +3049,20 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		if (reltuples == 0 || relpages < 0)
+			relpages = 0;
+
+		if (relallvisible > relpages)
+			relallvisible = relpages;
+
+		if (relallfrozen > relallvisible)
+			relallfrozen = relallvisible;
+
+		if (relpages > 0)
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..242155c423 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -669,8 +669,8 @@
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_freeze_max_age = 200000000	# maximum XID age before forced vacuum
 					# (change requires restart)
-- 
2.45.2

v1-0001-Add-relallfrozen-to-pg_class.patchapplication/octet-stream; name=v1-0001-Add-relallfrozen-to-pg_class.patchDownload
From 564851f3d0450c604732ecf468ce46457d36d989 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 4 Oct 2024 14:29:47 -0400
Subject: [PATCH v1 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.
---
 doc/src/sgml/catalogs.sgml              | 15 ++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 ++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 27 +++++++++++++++++--------
 src/backend/commands/analyze.c          | 12 +++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           |  6 ++++++
 src/backend/statistics/relation_stats.c |  9 +++++++++
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/pg_class.h          |  3 +++
 src/include/commands/vacuum.h           |  1 +
 11 files changed, 81 insertions(+), 18 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 964c819a02..174ef29ed6 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,21 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's
+       visibility map.  This is only an estimate used for triggering autovacuums.
+       It is updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       Every all-frozen page must also be marked all-visible.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d82aa3d489..dff4cd08a9 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -303,7 +303,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -558,10 +559,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * Every block marked all-frozen in the VM must also be marked
+	 * all-visible.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -570,7 +578,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3100,7 +3109,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 0078a12f26..d150d2b9a7 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -920,6 +920,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -990,6 +991,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 12822d0b14..2386009a1d 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2795,8 +2795,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2841,8 +2841,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2888,12 +2888,18 @@ index_update_stats(Relation rel,
 	if (reltuples >= 0 && !IsBinaryUpgrade)
 	{
 		BlockNumber relpages = RelationGetNumberOfBlocks(rel);
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
+		/* don't bother for indexes */
 		if (rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
-		else					/* don't bother for indexes */
-			relallvisible = 0;
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* Every all-frozen page must also be set all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 
 		if (rd_rel->relpages != (int32) relpages)
 		{
@@ -2910,6 +2916,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 38fb4c3ef2..0928592272 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 78f96789b0..4376355066 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1224,6 +1224,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1236,6 +1237,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ac8f5d9c25..7ed4df509c 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1459,6 +1460,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 1a6d1640c3..1b1f02546f 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -160,6 +160,15 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 			replaces[ncols] = Anum_pg_class_relallvisible;
 			values[ncols] = Int32GetDatum(relallvisible);
 			ncols++;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * NULL. It will be updated the next time these fields are
+			 * updated.
+			 */
+			replaces[ncols] = Anum_pg_class_relallfrozen;
+			nulls[ncols] = true;
 		}
 	}
 
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index c326f687eb..4bd9e10310 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1936,6 +1936,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 	relation->rd_rel->relam = HEAP_TABLE_AM_OID;
@@ -3931,6 +3932,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 0fc2c093b0..659ee68620 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 759f9a87d3..c3fd2919e6 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.45.2

#2Melanie Plageman
melanieplageman@gmail.com
In reply to: Melanie Plageman (#1)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Oct 22, 2024 at 3:12 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

The attached patch does this.

I realized that I broke relation_statistics_update(). Attached v2 is fixed.

I've estimated the unfrozen percentage of the table by adding a new
field to pg_class, relallfrozen, which is updated in the same places
as relallvisible.

While my relallfrozen column correctly appears in pg_class, I noticed
that it seems like catalog/pg_class_d.h did not have my column added
(this file is auto-generated), despite my adding relallfrozen to
catalog/pg_class.h. Is there something else I have to do when adding a
new column to pg_class?

At the end of the benchmark, the patched version of Postgres had
emitted twice as many FPIs as master.

This was meant to say the reverse -- _master_ did twice as many FPIs
as the patch

- Melanie

Attachments:

v2-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v2-0001-Add-relallfrozen-to-pg_class.patchDownload
From 1bd8bbf4a62013e52180d17bc065eea826f784ff Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 4 Oct 2024 14:29:47 -0400
Subject: [PATCH v2 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.
---
 doc/src/sgml/catalogs.sgml              | 15 ++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 ++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 27 +++++++++++++++++--------
 src/backend/commands/analyze.c          | 12 +++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           |  6 ++++++
 src/backend/statistics/relation_stats.c | 20 +++++++++++++++---
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/catversion.h        |  2 +-
 src/include/catalog/pg_class.h          |  3 +++
 src/include/commands/vacuum.h           |  1 +
 12 files changed, 90 insertions(+), 22 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 964c819a02d..174ef29ed6d 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,21 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's
+       visibility map.  This is only an estimate used for triggering autovacuums.
+       It is updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       Every all-frozen page must also be marked all-visible.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d82aa3d4896..dff4cd08a99 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -303,7 +303,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -558,10 +559,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * Every block marked all-frozen in the VM must also be marked
+	 * all-visible.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -570,7 +578,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3100,7 +3109,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 0078a12f26e..d150d2b9a72 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -920,6 +920,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -990,6 +991,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 12822d0b140..2386009a1d7 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2795,8 +2795,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2841,8 +2841,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2888,12 +2888,18 @@ index_update_stats(Relation rel,
 	if (reltuples >= 0 && !IsBinaryUpgrade)
 	{
 		BlockNumber relpages = RelationGetNumberOfBlocks(rel);
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
+		/* don't bother for indexes */
 		if (rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
-		else					/* don't bother for indexes */
-			relallvisible = 0;
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* Every all-frozen page must also be set all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 
 		if (rd_rel->relpages != (int32) relpages)
 		{
@@ -2910,6 +2916,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 38fb4c3ef23..0928592272e 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 78f96789b0e..43763550668 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1224,6 +1224,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1236,6 +1237,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ac8f5d9c259..7ed4df509c5 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1459,6 +1460,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index b1eb8a9bbaf..bc7092dc1ca 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -54,6 +54,10 @@ static bool relation_statistics_update(FunctionCallInfo fcinfo, int elevel);
 
 /*
  * Internal function for modifying statistics for a relation.
+ *
+ * Up to four pg_class columns may be updated even though only three relation
+ * statistics may be modified; relallfrozen is always set to -1 when
+ * relallvisible is updated manually.
  */
 static bool
 relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
@@ -62,9 +66,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 	Relation	crel;
 	HeapTuple	ctup;
 	Form_pg_class pgcform;
-	int			replaces[3] = {0};
-	Datum		values[3] = {0};
-	bool		nulls[3] = {0};
+	int			replaces[4] = {0};
+	Datum		values[4] = {0};
+	bool		nulls[4] = {0};
 	int			ncols = 0;
 	TupleDesc	tupdesc;
 	HeapTuple	newtup;
@@ -160,6 +164,16 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 			replaces[ncols] = Anum_pg_class_relallvisible;
 			values[ncols] = Int32GetDatum(relallvisible);
 			ncols++;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * -1, or unknown. It will be updated the next time these fields
+			 *  are updated.
+			 */
+			replaces[ncols] = Anum_pg_class_relallfrozen;
+			values[ncols] = Int32GetDatum(-1);
+			ncols++;
 		}
 	}
 
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index c326f687eb4..4bd9e10310c 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1936,6 +1936,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 	relation->rd_rel->relam = HEAP_TABLE_AM_OID;
@@ -3931,6 +3932,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 391bf04bf5d..6d241f449bb 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -57,6 +57,6 @@
  */
 
 /*							yyyymmddN */
-#define CATALOG_VERSION_NO	202410222
+#define CATALOG_VERSION_NO	202410223
 
 #endif
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 0fc2c093b0d..b915bef9aa5 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date; -1 means "unknown") */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 759f9a87d38..c3fd2919e64 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.34.1

v2-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchtext/x-patch; charset=US-ASCII; name=v2-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From e29e41ac11b614aafb58412e042c50f37e33ef06 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 4 Oct 2024 17:06:04 -0400
Subject: [PATCH v2 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.
---
 doc/src/sgml/config.sgml                      |  4 +--
 src/backend/postmaster/autovacuum.c           | 26 +++++++++++++++++--
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 934ef5e4691..a303f06e4c9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8683,10 +8683,10 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
       </term>
       <listitem>
        <para>
-        Specifies a fraction of the table size to add to
+        Specifies a fraction of the active (unfrozen) table size to add to
         <varname>autovacuum_vacuum_insert_threshold</varname>
         when deciding whether to trigger a <command>VACUUM</command>.
-        The default is 0.2 (20% of table size).
+        The default is 0.2 (20% of active table size).
         This parameter can only be set in the <filename>postgresql.conf</filename>
         file or on the server command line;
         but the setting can be overridden for individual tables by
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index dc3cf87abab..9bd96794ae9 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2922,7 +2922,12 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
+
+	/* From pg_class */
+	float4		reltuples;
+	int32		relpages;
+	int32		relallfrozen;
+	int32		relallvisible;
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3030,6 +3035,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
+		float4		pcnt_unfrozen = 1;
+
+		relpages = classForm->relpages;
+		relallfrozen = classForm->relallfrozen;
+		relallvisible = classForm->relallvisible;
 		reltuples = classForm->reltuples;
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
@@ -3039,8 +3049,20 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		if (reltuples == 0 || relpages < 0)
+			relpages = 0;
+
+		if (relallvisible > relpages)
+			relallvisible = relpages;
+
+		if (relallfrozen > relallvisible)
+			relallfrozen = relallvisible;
+
+		if (relpages > 0)
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a2..242155c4230 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -669,8 +669,8 @@
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_freeze_max_age = 200000000	# maximum XID age before forced vacuum
 					# (change requires restart)
-- 
2.34.1

#3Greg Sabino Mullane
htamfids@gmail.com
In reply to: Melanie Plageman (#2)
Re: Trigger more frequent autovacuums of heavy insert tables

I really appreciate all the work to make vacuum better. Anything that helps
our problem of autovacuum not scaling well for large tables is a win.

I'm not overly familiar with this part of the code base, but here are some
questions/ideas:

+       /*
+        * Every block marked all-frozen in the VM must also be marked
+        * all-visible.
+        */
+       if (new_rel_allfrozen > new_rel_allvisible)
+               new_rel_allfrozen = new_rel_allvisible;
+

Maybe tweak either the comment, or the code, as I read that comment as
meaning:

if (new_rel_allfrozen > new_rel_allvisible)
new_ral_allvisible = new_rel_allfrozen;

+                       /*
+                        * If we are modifying relallvisible manually, it
is not clear
+                        * what relallfrozen value would make sense.
Therefore, set it to
+                        * -1, or unknown. It will be updated the next time
these fields
+                        *  are updated.
+                        */
+                       replaces[ncols] = Anum_pg_class_relallfrozen;
+                       values[ncols] = Int32GetDatum(-1);

Do we need some extra checks later on when we are actually using this to
prevent negative numbers in the calculations? It's only going to make
pcnt_unfrozen something like 1.0001 but still might want to skip that.

In autovacuum.c, seems we could simplify some of the logic there to this?:

if (relpages > 0 && reltuples > 0) {

relallfrozen = classForm->relallfrozen;
relallvisible = classForm->relallvisible;

if (relallvisible > relpages)
relallvisible = relpages;

if (relallfrozen > relallvisible)
relallfrozen = relallvisible;

pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);

}
vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor *
reltuples * pcnt_unfrozen;

Again, I'm not clear under what circumstances will relallvisible > relpages?

Cheers,
Greg

#4Melanie Plageman
melanieplageman@gmail.com
In reply to: Greg Sabino Mullane (#3)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

Thanks for the review!

On Thu, Oct 24, 2024 at 3:51 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:

I really appreciate all the work to make vacuum better. Anything that helps our problem of autovacuum not scaling well for large tables is a win.

I'm not overly familiar with this part of the code base, but here are some questions/ideas:

+       /*
+        * Every block marked all-frozen in the VM must also be marked
+        * all-visible.
+        */
+       if (new_rel_allfrozen > new_rel_allvisible)
+               new_rel_allfrozen = new_rel_allvisible;
+

Maybe tweak either the comment, or the code, as I read that comment as meaning:

if (new_rel_allfrozen > new_rel_allvisible)
new_ral_allvisible = new_rel_allfrozen;

I've updated it. An all-frozen block must also be all-visible. But not
all-visible blocks are all-frozen

+                       /*
+                        * If we are modifying relallvisible manually, it is not clear
+                        * what relallfrozen value would make sense. Therefore, set it to
+                        * -1, or unknown. It will be updated the next time these fields
+                        *  are updated.
+                        */
+                       replaces[ncols] = Anum_pg_class_relallfrozen;
+                       values[ncols] = Int32GetDatum(-1);

Do we need some extra checks later on when we are actually using this to prevent negative numbers in the calculations? It's only going to make pcnt_unfrozen something like 1.0001 but still might want to skip that.

Great point! I've added this

In autovacuum.c, seems we could simplify some of the logic there to this?:

if (relpages > 0 && reltuples > 0) {

relallfrozen = classForm->relallfrozen;
relallvisible = classForm->relallvisible;

if (relallvisible > relpages)
relallvisible = relpages;

if (relallfrozen > relallvisible)
relallfrozen = relallvisible;

pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);

}
vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples * pcnt_unfrozen;

I've done something similar to this in attached v2.

Again, I'm not clear under what circumstances will relallvisible > relpages?

I think this is mostly if someone manually updated the relation stats,
so we clamp it for safety.

- Melanie

Attachments:

v3-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchtext/x-patch; charset=US-ASCII; name=v3-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From c2e29150e923f9782ce24a7a4e7d6f2d7445b543 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 4 Oct 2024 17:06:04 -0400
Subject: [PATCH v3 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Author: Melanie Plageman
Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      |  4 +-
 src/backend/postmaster/autovacuum.c           | 37 ++++++++++++++++++-
 src/backend/utils/misc/postgresql.conf.sample |  4 +-
 3 files changed, 39 insertions(+), 6 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index dc401087dc6..3bd22f7f1e7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8710,10 +8710,10 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
       </term>
       <listitem>
        <para>
-        Specifies a fraction of the table size to add to
+        Specifies a fraction of the active (unfrozen) table size to add to
         <varname>autovacuum_vacuum_insert_threshold</varname>
         when deciding whether to trigger a <command>VACUUM</command>.
-        The default is 0.2 (20% of table size).
+        The default is 0.2 (20% of active table size).
         This parameter can only be set in the <filename>postgresql.conf</filename>
         file or on the server command line;
         but the setting can be overridden for individual tables by
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index dc3cf87abab..364b46f672d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2922,7 +2922,12 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
+
+	/* From pg_class */
+	float4		reltuples;
+	int32		relpages;
+	int32		relallfrozen;
+	int32		relallvisible;
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3030,6 +3035,10 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
+		float4		pcnt_unfrozen = 1;
+
+		relpages = classForm->relpages;
+		relallfrozen = classForm->relallfrozen;
 		reltuples = classForm->reltuples;
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
@@ -3039,8 +3048,32 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If the table has been vacuumed and we have reliable data for
+		 * relallfrozen and relallvisible, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 *
+		 * If relallfrozen is -1, that means relallvisible was updated
+		 * manually and we can't rely on relallfrozen.
+		 */
+		if (relpages > 0 && reltuples > 0 && relallfrozen > -1)
+		{
+			relallvisible = classForm->relallvisible;
+
+			if (relallvisible > relpages)
+				relallvisible = relpages;
+
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39a3ac23127..a66f16e838a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -670,8 +670,8 @@
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_freeze_max_age = 200000000	# maximum XID age before forced vacuum
 					# (change requires restart)
-- 
2.34.1

v3-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v3-0001-Add-relallfrozen-to-pg_class.patchDownload
From caab36b29b9a9e2582283279d2e9b0fa404307ae Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Fri, 25 Oct 2024 10:43:39 -0400
Subject: [PATCH v3 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Author: Melanie Plageman
Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/catalogs.sgml              | 20 ++++++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 ++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 27 +++++++++++++++++--------
 src/backend/commands/analyze.c          | 12 +++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           |  6 ++++++
 src/backend/statistics/relation_stats.c | 20 +++++++++++++++---
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/catversion.h        |  2 +-
 src/include/catalog/pg_class.h          |  3 +++
 src/include/commands/vacuum.h           |  1 +
 12 files changed, 95 insertions(+), 22 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 964c819a02d..392a1f71eb0 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,26 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>,
+       and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       <structfield>relallfrozen</structfield> must be less than or equal to
+       <strutfield>relallvisible</structfield> as an all-frozen page must be
+       all-visible. If <structfield>relallvisible</structfield> was updated
+       manually, <structfield>relallfrozen</structfield> will be -1 until the
+       next time they are updated.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index d82aa3d4896..776d1673bbb 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -303,7 +303,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -558,10 +559,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -570,7 +578,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3100,7 +3109,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 0078a12f26e..d150d2b9a72 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -920,6 +920,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -990,6 +991,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 9162b9f81a2..6f1bc159ecd 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2795,8 +2795,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2841,8 +2841,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2888,12 +2888,18 @@ index_update_stats(Relation rel,
 	if (reltuples >= 0 && !IsBinaryUpgrade)
 	{
 		BlockNumber relpages = RelationGetNumberOfBlocks(rel);
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
+		/* don't bother for indexes */
 		if (rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
-		else					/* don't bother for indexes */
-			relallvisible = 0;
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* Every all-frozen page must also be set all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 
 		if (rd_rel->relpages != (int32) relpages)
 		{
@@ -2910,6 +2916,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 38fb4c3ef23..0928592272e 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 78f96789b0e..43763550668 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1224,6 +1224,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1236,6 +1237,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index ac8f5d9c259..7ed4df509c5 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1459,6 +1460,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 5a2aabc921e..69892f19978 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -54,6 +54,10 @@ static bool relation_statistics_update(FunctionCallInfo fcinfo, int elevel);
 
 /*
  * Internal function for modifying statistics for a relation.
+ *
+ * Up to four pg_class columns may be updated even though only three relation
+ * statistics may be modified; relallfrozen is always set to -1 when
+ * relallvisible is updated manually.
  */
 static bool
 relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
@@ -62,9 +66,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 	Relation	crel;
 	HeapTuple	ctup;
 	Form_pg_class pgcform;
-	int			replaces[3] = {0};
-	Datum		values[3] = {0};
-	bool		nulls[3] = {0};
+	int			replaces[4] = {0};
+	Datum		values[4] = {0};
+	bool		nulls[4] = {0};
 	int			ncols = 0;
 	TupleDesc	tupdesc;
 	bool		result = true;
@@ -154,6 +158,16 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 			replaces[ncols] = Anum_pg_class_relallvisible;
 			values[ncols] = Int32GetDatum(relallvisible);
 			ncols++;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * -1, or unknown. It will be updated the next time these fields
+			 *  are updated.
+			 */
+			replaces[ncols] = Anum_pg_class_relallfrozen;
+			values[ncols] = Int32GetDatum(-1);
+			ncols++;
 		}
 	}
 
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index c326f687eb4..4bd9e10310c 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1936,6 +1936,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 	relation->rd_rel->relam = HEAP_TABLE_AM_OID;
@@ -3931,6 +3932,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index fc1c125d0d0..21dfcb16d27 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -57,6 +57,6 @@
  */
 
 /*							yyyymmddN */
-#define CATALOG_VERSION_NO	202410242
+#define CATALOG_VERSION_NO	202410251
 
 #endif
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 0fc2c093b0d..b915bef9aa5 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date; -1 means "unknown") */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 759f9a87d38..c3fd2919e64 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.34.1

#5Melanie Plageman
melanieplageman@gmail.com
In reply to: Melanie Plageman (#4)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Oct 25, 2024 at 11:14 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I've done something similar to this in attached v2.

This needed a rebase. See attached v4.

- Melanie

Attachments:

v4-0001-Add-relallfrozen-to-pg_class.patchapplication/octet-stream; name=v4-0001-Add-relallfrozen-to-pg_class.patchDownload
From 2ed8278b4fb44050c4081b021bf59bcc33fd107e Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:19:57 -0500
Subject: [PATCH v4 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/catalogs.sgml              | 20 ++++++++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 +++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 22 +++++++++++++++++-----
 src/backend/commands/analyze.c          | 12 ++++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           | 15 ++++++++++++++-
 src/backend/statistics/relation_stats.c | 23 ++++++++++++++++++++---
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/pg_class.h          |  8 ++++++++
 src/include/commands/vacuum.h           |  1 +
 11 files changed, 108 insertions(+), 19 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index d3036c5ba9d..d218d3702e9 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2064,6 +2064,26 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>,
+       and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       <structfield>relallfrozen</structfield> must be less than or equal to
+       <strutfield>relallvisible</structfield> as an all-frozen page must be
+       all-visible. If <structfield>relallvisible</structfield> was updated
+       manually, <structfield>relallfrozen</structfield> will be -1 until the
+       next time they are updated.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 5b0e790e121..ec73471c2c4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -370,7 +370,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -629,10 +630,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -641,7 +649,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3248,7 +3257,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 57ef466acce..5ab75d5c977 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -913,6 +913,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -983,6 +984,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 7377912b41e..ee72a1e5e41 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2792,8 +2792,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2811,6 +2811,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2850,7 +2851,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2875,8 +2882,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2923,6 +2930,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 2a7769b1fd1..927b35ead34 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -628,12 +628,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -644,6 +643,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -660,7 +660,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -676,7 +676,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index e6745e6145c..376ce2e489a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1441,8 +1442,15 @@ vac_update_relstats(Relation relation,
 			 relid);
 	pgcform = (Form_pg_class) GETSTRUCT(ctup);
 
-	/* Apply statistical updates, if any, to copied tuple */
+	/*
+	 * An all-frozen block must be marked all-visible in the VM. While callers
+	 * are likely to check this themselves, it is worth ensuring we don't put
+	 * incorrect information in pg_class.
+	 */
+	if (num_all_frozen_pages > num_all_visible_pages)
+		num_all_frozen_pages = num_all_visible_pages;
 
+	/* Apply statistical updates, if any, to copied tuple */
 	dirty = false;
 	if (pgcform->relpages != (int32) num_pages)
 	{
@@ -1459,6 +1467,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 046661d7c3f..758a56f8a46 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -56,6 +56,10 @@ static bool relation_statistics_update(FunctionCallInfo fcinfo, int elevel,
 
 /*
  * Internal function for modifying statistics for a relation.
+ *
+ * Up to four pg_class columns may be updated even though only three relation
+ * statistics may be modified; relallfrozen is always set to -1 when
+ * relallvisible is updated manually.
  */
 static bool
 relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
@@ -167,6 +171,14 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		if (update_relallvisible && pgcform->relallvisible != relallvisible)
 		{
 			pgcform->relallvisible = relallvisible;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * -1 ("unknown"). It will be updated the next time these fields
+			 * are updated.
+			 */
+			pgcform->relallfrozen = -1;
 			dirty = true;
 		}
 
@@ -182,9 +194,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		TupleDesc	tupdesc = RelationGetDescr(crel);
 		HeapTuple	ctup;
 		Form_pg_class pgcform;
-		int			replaces[3] = {0};
-		Datum		values[3] = {0};
-		bool		nulls[3] = {0};
+		int			replaces[4] = {0};
+		Datum		values[4] = {0};
+		bool		nulls[4] = {0};
 		int			nreplaces = 0;
 
 		ctup = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
@@ -217,6 +229,11 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			replaces[nreplaces] = Anum_pg_class_relallvisible;
 			values[nreplaces] = Int32GetDatum(relallvisible);
 			nreplaces++;
+
+			/* See comment above in-place update of relallfrozen */
+			replaces[nreplaces] = Anum_pg_class_relallfrozen;
+			values[nreplaces] = Int32GetDatum(-1);
+			nreplaces++;
 		}
 
 		if (nreplaces > 0)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 43219a9629c..efd6bcb3860 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1925,6 +1925,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3882,6 +3883,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fbf42b0e195 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,14 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/*
+	 * # of all-frozen blocks (not always up-to-date)
+	 * Starts as 0 (if it has never been updated).
+	 * If relallvisible is manually updated, relallfrozen will be
+	 * -1, meaning "unknown"
+	 */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 12d0b61950d..b2a678df09e 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.45.2

v4-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchapplication/octet-stream; name=v4-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From 3a3ca54a118879d13c18fc2b266ee88edcc338a8 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v4 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++-----
 src/backend/postmaster/autovacuum.c           | 32 +++++++++++++++++--
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 39 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a8866292d46..84f2b44ec97 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8651,14 +8651,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is 0.2 (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 0ab921a169b..2d4a7778561 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2929,7 +2929,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3037,7 +3036,12 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+		int32		relallvisible = classForm->relallvisible;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3046,8 +3050,30 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If the table has been vacuumed and we have reliable data for
+		 * relallfrozen and relallvisible, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 *
+		 * If relallfrozen is -1, that means relallvisible was updated
+		 * manually and we can't rely on relallfrozen.
+		 */
+		if (relpages > 0 && reltuples > 0 && relallfrozen > -1)
+		{
+			if (relallvisible > relpages)
+				relallvisible = relpages;
+
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 079efa1baa7..1b10f53c659 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -667,8 +667,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_freeze_max_age = 200000000	# maximum XID age before forced vacuum
 					# (change requires restart)
-- 
2.45.2

#6Melanie Plageman
melanieplageman@gmail.com
In reply to: Melanie Plageman (#5)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Thu, Jan 16, 2025 at 4:43 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Fri, Oct 25, 2024 at 11:14 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I've done something similar to this in attached v2.

This needed a rebase. See attached v4.

Whoops -- docs didn't build. Attached v5.

- Melanie

Attachments:

v5-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchapplication/octet-stream; name=v5-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From c27e370df14f6f20d21a76c9563d6f00019949e1 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v5 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++-----
 src/backend/postmaster/autovacuum.c           | 32 +++++++++++++++++--
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 39 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a8866292d46..84f2b44ec97 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8651,14 +8651,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is 0.2 (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 0ab921a169b..2d4a7778561 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2929,7 +2929,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3037,7 +3036,12 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+		int32		relallvisible = classForm->relallvisible;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3046,8 +3050,30 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If the table has been vacuumed and we have reliable data for
+		 * relallfrozen and relallvisible, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 *
+		 * If relallfrozen is -1, that means relallvisible was updated
+		 * manually and we can't rely on relallfrozen.
+		 */
+		if (relpages > 0 && reltuples > 0 && relallfrozen > -1)
+		{
+			if (relallvisible > relpages)
+				relallvisible = relpages;
+
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 079efa1baa7..1b10f53c659 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -667,8 +667,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_freeze_max_age = 200000000	# maximum XID age before forced vacuum
 					# (change requires restart)
-- 
2.45.2

v5-0001-Add-relallfrozen-to-pg_class.patchapplication/octet-stream; name=v5-0001-Add-relallfrozen-to-pg_class.patchDownload
From 61e40cbe7c9df615461174301984552e31006a09 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:19:57 -0500
Subject: [PATCH v5 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/catalogs.sgml              | 20 ++++++++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 +++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 22 +++++++++++++++++-----
 src/backend/commands/analyze.c          | 12 ++++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           | 15 ++++++++++++++-
 src/backend/statistics/relation_stats.c | 23 ++++++++++++++++++++---
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/pg_class.h          |  8 ++++++++
 src/include/commands/vacuum.h           |  1 +
 11 files changed, 108 insertions(+), 19 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index d3036c5ba9d..bc9a735e5a7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2064,6 +2064,26 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>,
+       and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       <structfield>relallfrozen</structfield> must be less than or equal to
+       <structfield>relallvisible</structfield> as an all-frozen page must be
+       all-visible. If <structfield>relallvisible</structfield> was updated
+       manually, <structfield>relallfrozen</structfield> will be -1 until the
+       next time they are updated.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 5b0e790e121..ec73471c2c4 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -370,7 +370,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -629,10 +630,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -641,7 +649,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3248,7 +3257,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 57ef466acce..5ab75d5c977 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -913,6 +913,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -983,6 +984,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 7377912b41e..ee72a1e5e41 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2792,8 +2792,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2811,6 +2811,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2850,7 +2851,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2875,8 +2882,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2923,6 +2930,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index 2a7769b1fd1..927b35ead34 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -628,12 +628,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -644,6 +643,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -660,7 +660,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -676,7 +676,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index e6745e6145c..376ce2e489a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1441,8 +1442,15 @@ vac_update_relstats(Relation relation,
 			 relid);
 	pgcform = (Form_pg_class) GETSTRUCT(ctup);
 
-	/* Apply statistical updates, if any, to copied tuple */
+	/*
+	 * An all-frozen block must be marked all-visible in the VM. While callers
+	 * are likely to check this themselves, it is worth ensuring we don't put
+	 * incorrect information in pg_class.
+	 */
+	if (num_all_frozen_pages > num_all_visible_pages)
+		num_all_frozen_pages = num_all_visible_pages;
 
+	/* Apply statistical updates, if any, to copied tuple */
 	dirty = false;
 	if (pgcform->relpages != (int32) num_pages)
 	{
@@ -1459,6 +1467,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 046661d7c3f..758a56f8a46 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -56,6 +56,10 @@ static bool relation_statistics_update(FunctionCallInfo fcinfo, int elevel,
 
 /*
  * Internal function for modifying statistics for a relation.
+ *
+ * Up to four pg_class columns may be updated even though only three relation
+ * statistics may be modified; relallfrozen is always set to -1 when
+ * relallvisible is updated manually.
  */
 static bool
 relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
@@ -167,6 +171,14 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		if (update_relallvisible && pgcform->relallvisible != relallvisible)
 		{
 			pgcform->relallvisible = relallvisible;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * -1 ("unknown"). It will be updated the next time these fields
+			 * are updated.
+			 */
+			pgcform->relallfrozen = -1;
 			dirty = true;
 		}
 
@@ -182,9 +194,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		TupleDesc	tupdesc = RelationGetDescr(crel);
 		HeapTuple	ctup;
 		Form_pg_class pgcform;
-		int			replaces[3] = {0};
-		Datum		values[3] = {0};
-		bool		nulls[3] = {0};
+		int			replaces[4] = {0};
+		Datum		values[4] = {0};
+		bool		nulls[4] = {0};
 		int			nreplaces = 0;
 
 		ctup = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
@@ -217,6 +229,11 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			replaces[nreplaces] = Anum_pg_class_relallvisible;
 			values[nreplaces] = Int32GetDatum(relallvisible);
 			nreplaces++;
+
+			/* See comment above in-place update of relallfrozen */
+			replaces[nreplaces] = Anum_pg_class_relallfrozen;
+			values[nreplaces] = Int32GetDatum(-1);
+			nreplaces++;
 		}
 
 		if (nreplaces > 0)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 43219a9629c..efd6bcb3860 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1925,6 +1925,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3882,6 +3883,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fbf42b0e195 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,14 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/*
+	 * # of all-frozen blocks (not always up-to-date)
+	 * Starts as 0 (if it has never been updated).
+	 * If relallvisible is manually updated, relallfrozen will be
+	 * -1, meaning "unknown"
+	 */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 12d0b61950d..b2a678df09e 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.45.2

#7Melanie Plageman
melanieplageman@gmail.com
In reply to: Melanie Plageman (#6)
1 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Thu, Jan 16, 2025 at 5:50 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Thu, Jan 16, 2025 at 4:43 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Fri, Oct 25, 2024 at 11:14 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I've done something similar to this in attached v2.

This needed a rebase. See attached v4.

Whoops -- docs didn't build. Attached v5.

Outside of the positive performance impact of vacuuming pages before
they go cold (detailed in my first email [1]/messages/by-id/CAAKRu_aj-P7YyBz_cPNwztz6ohP+vWis=iz3YcomkB3NpYA--w@mail.gmail.com), there is also a
substantial positive effect with this patch for large tables with
substantial cold regions: fewer anti-wraparound vacuums and more
frequent normal/aggressive vacuums

With the default vacuum settings, you often see an append-only table
devolve to _only_ anti-wraparound vacuums after the first aggressive
vacuum. I ran an insert-only workload for an hour (with 32 clients and
synchronous commit off to maximize the amount of data inserted) with
the default vacuum settings. On master, after the first aggressive
vacuum, we do only anti-wraparound vacuums (and only two of these are
triggered). With the patch, after the first aggressive vacuum, 10 more
vacuums are triggered -- none of which are anti-wraparound vacuums.

I attached a chart comparing the autovacuums triggered on master vs
with the patch.

Besides the performance benefit of spreading the freezing work over
more normal vacuums (thereby disrupting foreground workloads less),
anti-wraparound vacuums are not auto canceled by DDL -- making them
more of a nuisance to users.

[1]: /messages/by-id/CAAKRu_aj-P7YyBz_cPNwztz6ohP+vWis=iz3YcomkB3NpYA--w@mail.gmail.com

Attachments:

autovacuums.pngimage/png; name=autovacuums.pngDownload
�PNG


IHDR��gC�9tEXtSoftwareMatplotlib version3.9.1, https://matplotlib.org/��!	pHYs.#.#x�?vIDATx���y��u�?��5���B"���X$�.���@��w�V_�K)O������_.������R�)����	"���3,3�0����x��:30��a�s8���z/��Ncy�I*�N�r$/�4.
��)X��,rJ9���R�@N)� �`�S
�������Z�bE���K��K�.��i�;��u�b���������-[���,��K/�'�|r�1�x������N����z�Y`�SI`���K���������;�4��4}��8���3���C���z��i��y�����}�&��O��mK^=���$��
��71�$$���"��M:E�Q���W��_�NA�6T$��~�%�?�#&�+�P�`�.b����k�N�� ���Y8){.iq���d!Y�-�NP�`�.b��������L�OyI������N�%���,�T��X>7{�S�d�@}S���������m�N&�7X�X0){n�oD~A2Y��)��]��X��%�rA$��*��)�k
���$v�t:����N�����R�T���E*�J:
��X23bCE��,vg
�h���t�]�6���c���QUU�t$�H�RQXXeeeQVV���IG�����N%������+�N��E�b����q�����L:���������c���QZZ�����M�&
h�6-���_29 W��u�N�c���l�2�W�.a��51g����aC�Q�h��X}��������������d�����s�*�je������:�K&�JA���-Z�������h��y���FAAA�R���Auuul��!������<���3��[�..\{��g�	��d�����8�m�d�@�(��AI���r����T*{��g4k�,�T@c��I�(--��m�����c�����V�Z7n����
������#�����K:����kc���Yk���$D�.]"�Je��Y�&�D@C�pr���o29 �`�����g�EEE����5i�$JJJ��V�^�P�!�X������������e�B����'� ��e|
���X2=�j]�Z���d�\R�E��N����*k���4�4�JJJ���7F:�N(
�P���=��Q�2�(�S
�h0���7[+((H 	�����7[���-��[�IV�~���\S�E��N�7[K�R	$���~m����-��=w��L�5X���K"�f�u��L�5X������&%m�'�rM$`����C����d�@�$��X�0���#*V&��]�����N���IP����F<���5K�N���S��@��%�1X0I���Q��,��Z�tve}��(j�t
����4FMJ"��t
v���}t�) �`$��u�1?I:@2��@����R�@��w�}�J�2������#
�,rJ9���R�@N)� �
���	&�G}����������C�����m�����W_}5�L��V��V�Z�����
�����X]�zuL�<9>���X�tiTTTDYYY���q�A�>��S�{?o��U�������Sc���n��())�V�Z�^{�}��������;;b��9��[o���~��/�������C|��_�:���6n�����Y�f���c�����o�8��v�;��`���c���!C2�UW]W_}ul��1n�����o~}��f�JJJ�{��^\s�5QTT����U���������+���7;��m����������Q�i���C=�<�L���[QUU���:t�/�0~��D�V�jt�g�y��������'�����oso���c��aq��F�>}2��f�����o������������;g��{���V?_�~}�y������.>���-�I�Rq���W\����w���W_�G���c�������e������x >����3����,�X@��f�������?��V��]�6n���?~|<���Q\\3f����;n��Y�Y�xq����OL�4)~��_m3�����8��k���O>����*~������>�rH���p�
����4���k��������&��������q��z��7b��1s��m�K����[o�I'�'�xb<��CQZZZ��&N���
��52��% ��t�y��Y�W�:u����G�>}"???k�+��_|qDD,Z�(��)�J�R��G�8��C�G���u��������m�����l���8��g�8��������{�����1=��<xpL�2e����{���/�|�������~���zh������K�H�R��og��?�C�����I�&��W�0`@����QPP����C�n������;7�9�������;�����O����R��-
�����f����#F�W\�{��|�t�����+��;����}����� .����3gN��Q���/�:d�M�:5�?��x���3k�^zi�~��QXX��L���q�������c�����G��
�V�ZO<�D\s�51m����X�vm�92�y���W�[�.F���v������_t�f�V�Z&L���z*x�������C�<������o�9��e�]G}�V����Z}f���q�gDEEEfm��A���8�8���2�U�V��?W\qE|������o�������w���w7u�e���E�"???�?����{��w�����x���j|Pw
�hT����q���c�������&N�����~��_��~���>o��M�~��QQQ��{oDD���8��3b��)��Y�x���b��A�����W<�������2eJDD|��'���O��'���<|p��1#�u������7�o|�q�������O=�TDDL�81�y��8����x��^�e��e���:+���?n���C����C��?�y��7/�����8��##"6��O�>��j���*�<�������G�W\��B�����w���8���c��!��GED�]w���~x��5z��O?����x�����_��f���{u��Fe����Q�vI��m�\�(
��M:�V�q�[,������.���?FuuuDD���������>SRRW\qE�92����Oo��{����^TT����G��r�������{�Z�5m������.��[M�4�=z�*_m<��#1i���|���W^��s�;w�����q�d�����[������^z������K:@.�R����k���S�N�����n���������'�xb���������[�����u�����_}�{+**�������eG�z����������__�����_�t�I������7��lIII\~��5~�?
��F��_�b���>5���_���k_�Z���o�\�f�b�����s���U�����{��������oq_�N���x`�g���K���o���O8��h��U��8���3�_�zu����
-Z���[@�(H:@.|��5���M���������?�8""���ktf������=^}��x���b���Q^^k�����%K�D��m7[:th������#"�W��UTVV���^=z������^}��H���������k��Y��)Sjt��j�P?`����i=-J:�n+���]��J�����d��������5k��������n�u������[�b���t���{n�������y��q��w��Gyd|��_�C=4Z�n]���b��)Y��Q�b��Q;t��e�j��{��;���(��QI��EA-���}%rvk�,YC�������Vq�m���|�I�����Z��������o�1R�T����q�q��7�����������K���;W�\Y�}-Z���ou� ��v�f�W]�t�!C�D�>}b�=��f��Eqqq���e������?��Fo���??�p�t�M1q��������8qbL�81����6lX�z�����{�����+V��;���k����p��
��,�{��'b�����y��q��w��#������^��[�T*F�#F��>� ���;vl�����d����?������/��O>�
��[5QRR�5_r�%1l����G�;t�=X9���g����oc��5:�l��:���O����O\|���N���?���{.y��x��W3�V�Z_���c����Y�:��%{��G���c�8��#w���o����"6T&�jg����:`�����3�o��M�~��5>;y����!�JE����w��q���+����rJ,Y�$""-Z�����/�)�}�{��Y����:P;+D�����I'vD^��O?�4�����;���kt���<�~��z�4h�������^}��-�����[G:���;C���_|�������e�W�>RZ�F����F�_�������?Deee}D���/��Y��%K�����4k^�vm�����s���/3��1#�~��Z��uk�N;���'� 9Ihl:t�3g������'��+�e���<3��=zt������U�V[���u�����RS�]vY�}�����K.��;,���ju@DDq��6{%�j�}��/�O�)�� �;��Li�������/�;��s��/^'�pB�X���o\q���W�8��3��`�?���t�r�-Yk|�����7k~��'��������e��7�7�tSL�<9""�M��w\<��#��S���a��x����O>����5:���q��I�j+/���Yg��5�u�]q��g������W�Z��sO|��_��'FDD���k����������s��q���3�<K�.�l_uuu����q��G�c�=�Y/))��#Gn��v��������?���8p`�������_���;�Weee���������eee��q��E�~������i��m��O?�4����\�;w�s�9'�L�R����� ����G�|<��S�����?�������G�m�6V�X3g�����g��92z���G���[�-����+���������c���QZZk����3g����7;w�-�D����z��~���"���~;�~��-��9sf���^Yk�z��G}4N=��X�|yDD,_�<�������kb�=��:Diii�����%Kb���5����M@���?�1�o��f�����f��<�����{��?����p��X�p�V?/..�_��Wq�l��o}�[1i������#�N�)��!Cb��	1b���0aB�gK�,�%K�l�|*��.]���m YyIh�Z�l/��r\q�QVV��}}���|0z��h��I���������SO����ow�������nL�2e��W����c���q����W���h��]�8cD����x��7��'���C�n�;��������k��������^[���]C*�N����i�����_��<i�����o�������>�(k�g��QPPP�;aWPYY����)S������I�&��S�8��Cb�}��)o��93�N��g���+W�����Y�f��m��o���O�>������]�����s����K���"�5k{��G���+z�����I��"?� 7^�'b�����q������lGz��r$���(�C���7�w���w���w����:th�1�z��tX��,r� �
U:��Y�f��������+VD��M�U�V��g�8��C���(���X��|��x�����g��_|1�,Y������1l�����K�����;�f������P�t:�C��K^���}�{��C�8��s����6��""6l��=�X<8�>��(//�QR�][A��7�x#��_��z~~~t��1��o6l���g���+���������/���5�Ud�]��:h��e�92�
�
����g>��qc���+q��W�+���Y��7��s��Gy���}��q�e����IS�U{��W��g?��#GFqq���������c��1q�E��~���g���c��11d��Z���c�8��#w(;��"/�
����c���q�y�m���������;�����g��}����AP�UC��
�&M���L~~~�5*k��g����X�l��AY���Kc���	�H��z��U���V�\�@�]��z6������i�@�]CA�vw���J���[�h��I���;wn|��'QYY�[��v��E��mwFL��Q�U����?d��|��x����S�N�p���>�k��b���q�������� W`����z*^~����s�9���l���3�f�����/����:th�{����k�Z�P�G,��t
�1Z� ������,[�,.������O>9Poo����q�����>_��Ww���-���������wj`���'"��)�@C��TWW�7����7o^f���,n���Z����{��'�C��~��E�����4V�\3g��1c��]w��q���e����N�q����������w��G��i�
��?&��sRI�BV=���������Z��o]�t������x��'b��a������m���6m�D�����?�a\{��q���FuuuDD�X�"���o��	"�������by�	��]��u�k'��������e���Q���3����Z��O<�F{��������V�Z�%�\�Y������G�z��5~�.�ZD�&�hl�
"�q��I'�B�N���f�PED�s�9q�
7���_|q<�����K/e����?����.�(N;��Z��>}z�|��;�}`���_F��7�@C�k'����g�}v�����)��w�}w�R��d���~�U����/FUUU��������v����=yI��3&N;�������u�Q��CE~~~�r:4�lk��U�p����P
�v�o���������a��>�h4i�$�YJKK�U�VYk�/�i��Q�������q��W���x����SOEiii"�
��
6$�`k`����S��������g�z���>�l���%����*�.]����m�D�l��:�={vy���h���Z�������O�pj���QUU��

�C���v�f��T*��u�9�$	h`������#��y��e�:w�/��Bt��9�d��sO�<p��())I(
��)���e���QG3f����m�6������{��"�������N>��d�l��Z�jU{��1y���Z��-�������{��w������{�����g^|��8��Sb������;�w�����H���c#�Je~]}��IG�!Ih(�&L�Z��K�,����������V�Zm���������W\qE�v�i1|��8������,k��������;��#x������|�����~{����*@.(����c�n�v��W���1c�������g���q��������;w���[Giii�����9sb�����K�Rq��������:e�o
������������c����?�1�:��������m���1z��<xp4o�|������������+�O���
��$��H��9y�k��q��W��W^�t:f����O��s���+���2JKK�U�V��K�0`@�h�"'�vX��T*{��w����IG��NUUU�?>&M���-�-ZD�.]b���QVVV�{W�X�&M��S�����c�����e�h��]r�!��[���-v��+b��q�p��X�dITWWG��-�_�B������]�~c�����o��"???��k���{��o4D
������cc��!���������:6l����/��[n���ov�i�����}-~��_D���k�����?�p<��s��{�E:����=z��_���w���x���R�-��=:F���sc�����o�������_���v[L�0!6n����<����7������U�V������{.~����[o����{��7�xc�x����h����++V��!C��O~��-�_ED�[�.~�����O<������7��Mt�Aq�M7����Y~�����_���������{��?�08��9rd�?~��W�t:�y�����~����k���Q���c��j�UD��)Sb���q�����n��+H:@.TWW�i�����Zf�M�6��k������3gFeee�����8��S�����!C�l�����L����S�NQVVUUU�x���;wn��>� 
���^�k�n'|��3fL�r�)�b���>k��mt��1�4i��-�Y�fEuuu����O~7�|sfn��yt��5���c������]y����o�8��S����(�����?f��tP�r�-q���G*������W��>?���3Q���1r���2eJ�l�r�w7m�4N:��8�������]�n�g�������-����X�`ADD|��'���|'��-�����GD�{���^zif�[��V�u�Y[�����o�6k��8��S����6m������s��>}�d�_�zu�7.y��x������^~���9sfDD|�K_�k��6�����t:/��R\x�����f������b����}��M��(|V~u����c�=���Y�7k�,�?��2dH|�+_�E�E���T���?��o�}��w�qq�YgE�v���~�6m����n�q�q�QG��o�O<�D|���PEDy����B���#�YM�92�/_��;u��<�L���~[���Y�8�����������>���S�w>+�:��s����������S�T<8^~��8��2e`����'�|2N:��Z}/�aR�E�����X�t��WqYD*/�[��c�x���7+����={�����{lf��{������QVV����}��*C�V�����}��7���#"������n��V���s�=�����M�6�f���Z�n�[���{�������w��_}^��m��+��/�0����O+��FB�J���;OL:����F��J:����g?����ow�1�C��_|1""***�����~��;%G��=c��1~����x���w��[s���f��F��q�U]\s�5�,������U���;��[&`��h
��3������>;S�1v���V���{�L����������
b������� .���z{���,�9���m��ut��5���s����\l.��X�4��*�$4D�t�	��N�(|��_���[�x������7�|s�g>�����_��������?�h��(//����m�[�vmTTTDqqq����[o�����:��w>s�AE^^^���k�.S��r�����&�?����"��N:	��,�Q���_��w��5Z�h���1{��H���J�6��d��5jT����q��:�[�bE�`��1#k����N����kW�����������5�!�W$K�JqY���L:����,�[��M�Z�i��u����:�����,�K��1#�����|������[�l�����U�UTTT���s��tv+���NI�X4*����VI� 	%%%�>SZZ�5�^�:�k���q���oV~��g�8����W�^��s�(--����H�R�=7�|s<��s��T[�V����5kV�o�G�f_�NDQ���
�,�QX�vm���Y�&k��<�����i��e�����}���{�v����{j��.�7o�5�^�:'����vD�I��!*h���t
�!R�4
K�,���e��e~���-Zd�?�p��������k}w}j��u��h����@���$�II�)hL����&M�����gGyyyf���[�R��\]]&L��p@���""&O�\�<u��g�����������h���X�lY�����KY�����K�FUUUf���W���6mZ��?������T����=������������|�I���X@��a��x���k����c�|���g���P�_���w�q�5�QZZ�5�]���g��#���UUU�~`gS�4�]w]�Z�j���}��x��3sqqq�1"kO�6m��� 3�?>����{���k]@��u��y����:��g�7�tS������`gR�4.�3�<36l���=��O���:+k������-[f�����������o���m�?}��8�������t��-�5k��_x��X�|y��q�1h����n��8��ck\��l���8qb���X@���[���x���b�����K/E:��|�f������E�e���o?����x��EY�_~y\z��Y�#"�,Y��rK���?����T*z��U��yyyq�Gd�+V���7�xc<������;�����x��h��uf^�`A0 .������7��f��x��������n���c�=V���S�t�\8���b���������o���6m�D�n����2f��Yg�6m<�@�j�j�w~����;��3&N��t:n������~={���-[���Kc����q�����/�<���S�N�q������'�x"S�5m����O~���c�����g�u��5�����'�+V���������/~���/�]�v��c�h��I,]�4f�����5�PyI���������_��W2kK�.�w�y'>�������7o�<�Hy��[����0�����g�zuuuL�:5�x���>}zV��������j�����_���QXXX�������^���{o���E������	&����
�W
��F���,^|������c�=����&M�����|�A�p�	���k��1a������EEE[�w������>��rK�R�:���~S�L�+��2��:u����Z���O�x�����{��<p�g��������o�=~���)3�����t:���&O�������&M��}�������������z��u������cc��!���������:3WUU��q���������G�-b�=��!C�D��-��������W^�������+���8�t��zht��u�Q����Oc��q�������K��� Z�j={��8��-#?�����G|����\��o%���iGz��r�RAAA4(
���l��Yw�q;��\h��}�|��I������(� �`�S
��)X��,rJ9U�t��0x��H��I�`��@����R�@N$��g�+/�&������+��J:��,����
O]��"�$���tv�U��_Q7��H:��,h�:���yx�)hl
�5�J�6[K��	$���~m������"��$��]U����{F��?�k
�h0���6[��������d��q�fk[�����gDA��Sd��A��J��� ��m��5	���v���9???R�$�"X4(��5��W�Z�P�l�W���7�y��,�-Zd������������c���Yk
�`�`�����D~~~���y��`������s�F:��Z/--M(��
���J����,�-[�YK��1w��(**����Giii���G^�~7�~TWW��
���<������:�����oV���,�v��EUUU���g�WVVFeee,^�8�dM�6��;&viyI��J�R��S�h��E�Q�F�.]"???�(�K+H:��g%X�r����qc���F���4��o���IG�]�,�T*����v�����k���<V�^UUUIG�T*���QVVeee���`���R�(--������H��Q]]�t:�d��*///R�T�R���@����N*������c[��tX��,rJ9���R�@N)� �`�S
��)X��,r� �@�X�0���#��N:	��qC�	jF����&b��I����%��|�t�VX���OvA
�`7�N'���������Sl�����W/�h�>��J�.�z&�`�`@#�����{'�"��@����R�@N)� �`�S
��)X��,rJ9���R�@N)� �`�S
��)X��,rJ9���R�@N)� �`�S
�����4v���|N�I�����N����c�m�E���IG�������S��c�X	{�����N����������O?M:
��7��t�z��t��,]1���S�X'�������_A� A�tDzc�)h:�Q�)�P�6$��������G��L:��f�"z�J%�r�� �^uU�)�����I:B�R�����km�%��aK5m{��gI��"/�4.
��)X��,rJ9���R�@N)� �`�SIh���t��5+����7o^�X�"�6m�Z���={�!�EEE;��U�V�k����M����(..�n���a��:u��o�X��|��x�����g��_|1�,Y������1l�����K�����wg��W^ye�������o�y*���?<F�_��Ww�-����t���{��^t��!�=����_�������
6�c�=����>;������_������_<��[,���H��1v��<xp��'?�t:]��r� �
�o����������c�o�>6l��g���+Wf�������?�^x!�5kV�7������#���:k�m����K�X�hQ��??Sx�N���o�u����~��:|K����t���e��q�E��O>��/��s��[o����^,]�4���
�:���o�9��S�7f������������x��c��E���o���sc��)q�)�d����[����}G������k�����c��q��������7�������1c������������3fL�����+b��5���C��_~9����W�^��#�l���Q�����6_ '`�����c���q�y�Eqq�v�����w�����Z�����{v�������%37i�$���?F�-��?�J�������gfm��q���n�-�\S�UC��
�&M���L~~~�5*k��g����?��Q]]���<������6��O~������m���z6h���y����v��m�y��'����;�Fo�q�QZZ��'L�,�aR��P�U�Z�j�����+�����1}���\ZZ�vX���to:��'�|�i���z6������i���'N��5~��_��6�H��z��+�d���u�&M�lu��)S��>}����M�oz@�`��?��Y������S�N���t�R��6���}I+H:��������_~9k��s����E�e�{��g�����s��x��Z��q��v���"6nL:	�����r��X��cQ�vm�Q��������z�l�������N>��0`�6��^�:k.--�������aC�[�.�6mZ�{6�h��Z�iM�>}���~L���%�`�U��;1��C"�5]l��zP]]���7c��y����������{v�����Z�]\\��;w���;����G�����I:�SP�t`G,��/�_��v�s�_yI�]v�e���Og������.]�l�leee���I�Z��������Z����j]�	���{E���t
`GT��z������O:�
I���v�m��_�2km��Qq�g��|QQQ��~��Z��n�����M���k�3���N@DD���-"�J:	PWU�~f��Zk~���WZ�P"���i�~����O:�
�v�|0.�����s�9'n������Y������V***�{g]\t�Eq�i����������O����_��F|�[I��=T��5�5k����H��'�`��k'����g�}v�����)��w�}w�j��/mZV�f��Z��tAAA���-i��]�k�n������XE(����������vZTUUe��:��x���"���pz���y�����������m���<Pwk_=k.80�$�6X;��7�����Geeef����G}4�4iR��z���5��3�V�7������:P{���������J;,�4�6X;�?��Ow�q�z�����O=�T������M�>���Z��2e�6��G�{�E��2k���CJ�kS�UGS�N���:*�/_�Y���w<���QVVV�{8���y��	QUUU������6��G��qYs�^�"�u�����`�������#��E�e��w��?�|�m�v���w�}�_�Bf^�fM����5:�f����H�J���N��<@�Tl��*80�$�>X��p��8��#b��y����;�/��;w�)o><k���{jt�/�K�^�:3���?:u��S2���s�}DD�a�%�`����-[GuT��1#���m�x����{��;��s�=7R�Tf~���c��)�<SYY7�pC��y����2[�a����3'k�x������`���U���c����'g�Z�l�=�\���{����_�8���3������������-�O��q�%��G}�Y���G�{��;5�e��e�y-ZD�>}J��+H:@C1|���0aB����X�dI������]|p�j�j�{������?�k�����	&�W������[c����}��M��/�<���d��������V����x�����K_�T^^Biv}
�jh�����]y��u�k��1Y%V[���{�=��#G��t:���^2$��m]�v�E���y�2��?�A�v�iu��^��qYs�a�%��aP��;��3#�N�y�����������x��K/��n�)W���^�.*�~;k�x����4yI`�F��&M��#GFaa�V�}��_��c���7��T*�	�q[����^�>k��K_J(
@�P�t��"�N'�v�=���s�y�������G}�V��������k|��_���;'������57��'�[�L&@��i��E��I�>�b�������J�p�%�!���x����4
��h���Q5~��,��S�PG���5��lMz�J(
@�������5�<��-���OR��bXl�,�:�����w��Z+Q�P#
������#6l��B*E$�)H:�Z�$b�;*�NB]�7V��)������4�>�m2s��7cy�{�h����57�o��o�"�0
�,�Fl���/��,O:	;� "�%h�V�������!�
^���IGh0��@r����
��TlL:�nA@�)�h���N:��V��'���o�:�
�t�� ��:�[F��3�������z����j�4RM�$�hH
bmtK?��UG����4X]�D�� 
��#�(
�,2z�8��I����7����F�����=}4��|rr���KqU�!hd����������""�L(
��,h�*�����w����J5W�t�my��g��W_�%K�D�V�����N8!�6m�t4H\���g��&�j''X����g�}63s�1��s���������_�zL�4i��:v�w�}w{������q�����K(	�NN
�~����M7�eeeq��gnu��E�����O?�4��tDD�R���H���`��>|x����N8����.�j��X��GYk�&�j'/�<����2�#FDII�V�^z����'�DDv����������s��e���sr�5U��5�JJ���_L(
�N�`�X�"�N��)�:����w������F*��t:EEE�����Qz�u��}���P�BB$!� 	(h�D
��46�x�����n���-j��V�8��8��[[��AfH�PI �*$�����'����gW�~��j�����}_��(�\���������%�\�i��ua-_�<_����z��Y�j��i�R��T��M�����s�v�VE�3f����^zi:;;�����Ks�I'u�?��S��7�1��~{����~��|����k~���g���[�O�(	l��z_0o����q��e���/���W^���������5�WI���|�����������vo`�������rK�Z��`:�i�^��d��$��Z�=�e�[�zun���E�$y�������>��8�k������i��{��{R�][��z�a��mW���/������r�-���HY�I�c�=v�����d���]��E��))�kg������OZv���4���^�������������p�
]���
�k_����v��a]��V���������]3�>��$�}�^�5|��$IY�y���_�����wI��(2c��-���"����nH	���X�
��e�^���>�t=/[�,<��K�Y�xqf����(�$Gq��\�ti���a��'(��g��������8=M��>��455u�[���/�����f:;;S�e�����z��V�Z�'�|���	&�!5�L��g��MC�d������S���#G��#�LY�)�2���wr�9�������+��o~3_���
���g�|��/{�w��uV���U���/z�M�rH��������{V�����K��(��e���>;���[v�e�|�CJGGG��LQ��G>���������#F(��OY�IV���%�������:*���J���,�����;���q�[<�'?�I��HQy��X���S�6d�����)��7jHV�|���'?������,��weY��c��/~��477��W^ye,X���������z�uw��r�������(
l��F^���~6�������<>�`���3~��s�1���UW]�����]�	'�P�������]3��w����KEi`�5�+I�������������f7���c�����C��'j�:�u`��h�:@#u�O����N�xW���GI�YV�A��3>���y��EYs����eu�\�
�����QQ�1
��>������w�����TeH��V���\uU���V���4lX���1`�4U�Qn��/�_��i=��M��z���..�2<�@{���\�2���)�r��9�������tnL����=���wW��4����:l��`���w��������N{{����KM6l��$[GT��-K6n|a�(��HM���Y�����?�<zt�q��b��{�;3��3����aXO?�tN=��\s�5I��,����(R�e
�A�Jmsj����3��&K��������y��/��w���D@�t����WC
��-[��?<<�@Wq��%V�cG����i�&X��&
*���G?�������j���y�[��8 ���K
��(@�6�v7��4�k��e���K������Yg��O|�0`@�������'j��O�&
(������7�(�E��|�3���>U�k�,�[;�Lv���,$M�����O��e�����h����6�v�����</��X���O�E�W��Uimm���5^R�5��<��Xc���znii��u5:;�E�����RM�S��i��%I�����?^��j<�H���vm�,�J��k��)����$K�,�m��V�+�����G��YM�S��$9�����?��O7�J�$/-�7����!X�|�;s�g�,���7��G?��F\���`�W�P��`%�\����})�2_���r�G����k��@�fi�����qS���ZuQQ��w�����-�����p�
9���2r��t�A�u�]3p��m>����n��]����_k2zr5YxA�
��d�������n���e�$Y�lY~���o�YeY*���mn�<n�����,��aX��s��'g���I��(u5�G-�S;��ZMj5����������+W&y���,�F\
�Q7$���]��GhH�>���\�2EQ$I������N8!S�N�.���A�5"
�G<�P���vm��j�P��X>�`����E��,3j�����?���3�}5��-�S;�����&���}�M7��$)�2EQ�;����+��������T����{������G��w����� msj��S���K��k���I��(2y��EQ�+�>n������k
�z��`���]������uY8�v0$�eR%Q���z_0m��477g����?~:::��R�k�>ncG��b��Z��5k��h��7M2�k���t,ZV]��`���t�{����s�1�����������_�:�|������I~��d��m�9�~��M�_~!���c�?�q��?��477'I>���g��������������)�vf#W��:/����;,��~��������,^��W}��uI���S=IK���X���c�8�����@���$y�����.�,��
��7��)S�����t���?�,�F�����>����c�m\Uu��80��<3:��(@���K&O���\E����e�r�����s�M�~�2j��8p��-�"�<�Hw�v2�]���w�.�#��S����~���Oi@2������$T��h4(E��U����`��?����(�E�$)�2I�~��,Z�h��}��-088t��V]7;�6>�5�g����hi�?6���;97-�������vD��Y5s�!����!��s��	;TvP/��g�����W���hH����q
�6);:�~��5k
�����UY��?�\��f����*J�w(����Y�j���zUZF��(
@�����g���[g��(	@����^R�5}zEI�X@���hQ6��W���1`}����k���C3`�����--��������rnQ��w�[����[�&X9$EssEi���`��?HQ�zfY�
����i��3*J��4�kG�eY3ww������g�m����N�^Q���aX��Xm��K��������m�]w�\��f����*J��4�k��y��g���y���s������/�m����(��>�����n&L�P��@_�>{v�����<rdEi���`M�8q����5����3s��g��?�i������G�_��_��k��~����I��`�����+J�75U`k�|������2p������������+���Bk7-��1��$}S�)�J�i�����h�d��9���*N�6�|2�=V�6h�����M��+I���1IR�e.������W��M�g�����O�����4}S�+�7n\��k�$I{{{��������X�������#�^�W���]w��������0	����k�����]��k���]�k���0	��t>�l��~{�Z�����z]��+����(�$��#�
����#���/,EZ=��@}TK�������ttt$I���^{�Uq"�t�O~w^��
�=o�����l��k6�+��H���=���i��D�?�L�y��?���72�EX��������/~��)�"eYf��9������e�������]�IZ��Pe{{��k_��:}zihH���_�]�����x���y�����~��<eY&I�����������;���%��Nl��\�~K7�n��G68
I�
��8��E�Cg����,�����9��s�#����s����%�����2�]�� 
)�z��%V��(�����,s�A��?�iF���	����H�����_��/e�O~�5|�2��>P�`�G�l
}{����7����'W
��`mo����t�Ay�{������inn��hu3hd2n���~�ei^}g�������S�4 @uR�u�Ygm���(2p��6,'N����c��!@5:W����S�����+J�8=�`g�~�����/,��d��P] �i�:@_�>{v�<���M��A�hX���u����4�,�
�e�����+J�X
�*�����q���5X@_����g������~'V���Zq���__��g��Y���e��A3f�(���4VC
��8����F��(���������X���W���R����,y@��q��<{�=5k
����aX�[~UE��SO>�dn����|�����[r�m�e��U]�'N�����o��������y�2i��:��u������������}u��!Xg�u�6�Y�vm�~���z���;wn��J���g���=����[���n�y����o�9mmm��z���g��:(MV���zl����3'���'��_�2�<�H~��|���OKKC�����[�������;��M
���QQ�j4�Aj;L�:5?�����O}*��{n~��'I.������2dHV�^���x��9����i�n����9�Wvv���u����T�W`=����ln�����7�����w�q�����J�:4���3m��r�!�6mZ�������-�~���#s��Gw��@����t�XQ���kzUV��u�Y��o~��,s�Yg5������[������~ijj�y7o���fz����k���wO�=��(
@5�^��������Q��$�<�H���������{g���I������u����T�W�8M�0����;��0	��Y;kV��:cFEI��+���^���SOU�`�m\�"����fm�����N�+�������>�58��4[����k���8����T������+���+���v���0
�p�����e��59rdF��q��U�J��g��_��4
PQ�����Gy$��RE��L���Mo�8U��s�=�<yr�����w���[?������M�=��
�[�}����u����T��`m��1s��������F��Y��,SE�O��=�����u�l��,[�l��-Z��/�<�_~y:��\|��9���=�SO=���~z��<������J+�Ln�N������q��l\�4��X�`�(�Mc�bd�������%�O�<{��5��N���l=EC
�&O��]�����|��l��!I�������9_��W�-����;������/�8��rJ����o;g�}v��	��\��d���k�s?�uD��q��<����}���R�5��E��,����(��hnn�E]��;�S�<�G�������������=��#C�������c���n�E]���~�7������SO���c3s��
��\:;���
���6��o���?��iz��`=�(�m��|i��=��C��o|#��M��l=��~���r�)����K��1"#F����|������}(�>�l�d��������<���8p`������������e����}����68
@����	&ls�UQ8p`�
��'����q��8�N){����=[��?��?d�]w�)������$��O>�o}�[9��3�%�?��?��SN��=?�pN:��n��':���]6������3�H���km��'����
JG���7f�qO�u��j_E���������R�5��F\���|�������\|��]k�\rI�`�3&c�����v{�)�2F�7��b��5k���7�<rd�@��Tu���eW����x�����m��g���_��W�?`�D8���3�k.�2>�`������`�N�^Q�y`�d��c�������(	@���^���L�~�j�
6T�������x����A3fT�zX;�E������kEI�����k��#���*J=OC
�n���477w�\{���u�����3ZZZr���ws����'���j���s����]�`�zh�&������.�.� eY�,�L�6-oy�[���#�<2tP��Lggg.���nN��}�������s���U��(
@������+J=S��:;;��_�*EQ�(���=����N;��$IQ��/~�w
��w_�;�����N:��0}X��uYw�5k�3fT�z��`�s�=Y�|y��L����o�����_�e/^�x`�3�$w�uW����e�������c���U���Z[[��O|���uw��r���"��Z] ��Z�}�}����<b��L�<y���{��3b���X�"I2w������;t����������������f^�n]������1~������/Y_�bE>�����s���'��w���6mZF�]�]Y��3gN.���\x��y��gk������������j�=�f0eJ��
�(
�Lu/�Z�hQ��(������r�{��U����Ov����=�yO,X���-^�8�s�f��~����~��{�.]��.�(]tQ�d���=zt����W��'�����7���3���>��W����6-�j�>��$�s��k���]����3_|��������n���Y�x��6lX���o�=�yO�R�beY��k����@��T�����t��n9s��e]��
��3{�8 _��s���f��Q[�g����������?_�@�:<mm5k���W�z��z_����&I�����?�������n�yk����RE���4������.����x>���'I,X��z(�=�X�/_����80#G���q�r���f�]v�[���Y�j��Q������(
�\u/��o������_����:'�x�v�����&���O�E����{�3�d'N������Vh�=�fx�a]�����}���1c��(��e��~��;t��>����#F��C�����6-�4cFEI�g�{V��t�I)�2Ir��w�#��v����|$w�yg��(��t�I]eXP��������������+J=[C
�>��O���)�"eY��_�zN;���\�r���\�2������]g���/��o�V���u��~{����BSSZ9��@��5�k�=�������,���K/�4&L�?��\u�UY�dI��%K������?��L�81?���S�e���/�����������U38��4
RQ��Zu�Yg��9s��?��?�J�V�\�o}�[�����$)�"�
���kS�e��������w�+g�}v���+Z;{v��:cFEI��kj�e�����}�C)�2EQ�(�$�\�e�����^�:���]kI��K�3�<3?�����,��oR�5h����@�������|�k_����z��/)���'y��MozS����|��_Nsss#c�m�??/�YkU�/���K�����mo{[n���\}�������#�d��eY�jU��Q�F�U�zU���7��c��ATET��k_����d��[��c����`A�
��MI�YY���O��,J��E5���G���{70�.�`=o��i�6mZ�*w�W����v��������u�d�u��EQA"������?�=�W���k6��:cF��@���B�V���5���2p����7��g�QA"�=Z�������G���w�����vkV��w�M��g���C:^����d�������j��G���w�3-��RQ2�`Th���y����������o�����5�Nzs&]��uH���Xu�����Wuv$��]@5Y���KW�X��.�,�f���w��%K�d��Y�~�6�SE:::������K6�����ZM�Fjh���>�O|��������\�CY����c������O��&@#5������QG��s��,�E������b�x��f;���`��ZM�FkH�����w�3s�<��PE����q�2i����=�k����A��l����7/6l�z�${��W&L����u�����q���^�o�1EQ�(�L�4)�\sM�|���t�MI^(���W��Y�f�����3�<����79��R�e�d��E9��Ss�����k�mD|�n�vy�����q
��>�!X��w^��,��1"�]w]�<��W�7p��s�1�����\y��1bD�������/��~�c�M����_k���j�4Z��/^�{��'EQ�(�|����{����w�q�����,��y�����;�������w�/ij�&@������oN��e�$9��S�����m�!��3�<3I����O~�������6)�?��U�{��E������q��m��u��m������E��,����6K�,������#Yt_��8X@R��e��%I�����c7�M���������x�����I�&%I����Y��'(@�,y8�x�vm��j�T��X---]���7��
KY�I����W<s�]w�z�7o�&h������=�A#*�P��`�1��y�����f���]�?��+��f���>�m�����&@U�^����{'I����E�6����S��o���-��j��<��)�"I2t��nJ
�msk�qS��P��`���j���Y�x�K��6mZ��J����?���o�����o���#eY&I&N�����g��d�����S7�-����X���k��w��y��Y/��]�zW��HQ)�2�y�{�`���|w�W�3��L��H�477g�����������%��UM���4��c�9&<�@��W��U���w���8qb�?�����HQy��������c�����������o������,�$EQ�/��/3b��F����X���45W��*
)�z����o~��)�2W\qE����e��!5���������+W�(���������������,��;6_����lZ�5~j59����7��M9��3�n��$�C=��:���	&�7��M�;��,[�,EQ��~.�2{��G~��_d��q���-6nH?P�6~J5Y����$�������r�!y�����}.������������{N;��|�����#��.�z(���vm�,�jX��5jT������_�j��,^�8eYf��v��{�Yu<���6�v5)8��(��qX/���{*�v�`��RM����,�g+������nO��[���}m����t�^]�l=��������<��<������u�����O����*���m�5(������w��ei��U�h��F\r����_�j/^���������^-+2���n9����[�h��`��3'������{��o{�����_��Wu��2Y6�����E��)���!�g�r@��4�����\u�U����2|�������i������72�
�����%�:b�{6.Y�5W�����H�	�=[�V�3��L��@���W;vTssZ����8��n
�
+�*�2EQt=�X�"^xa.������>9���s���f��	�����sj��^����-�i�������]s1xp��u��toJ��
���q��y����|&�'ONY�IRS���C�S��T&O����:*�\rI��]��h�vj��k��jr��4�k������?��z(7�xc�8��>���,�tvv���������d��v�������k�mDD`tnL�[�6^[�!X/6c��\x��Y�pa.���w�qijz.FQI��,�z�����?��G�I�&����t~��F�6c������5Xl��`=o��y����+��2O>�d������HY�Ij��{���{���w�}��7�1]tQ�y����@��pn�<|\2xT5Y�}*+�z�1c��#�H�����u�]���?�1c�l����c���1����_��_T���9�������w�X/v����_�j�x��\y��y����t�a�e��,�n��\q����i�&X�`�
z\�����s�q����.���s�d��I��(*N}����'j�`�-zl��
><{��W��k�0��8��-�S;�Lv���,�N-U�����/?�����(mmmU������w�/i���������U,]�4������/�8w�qG��,�$IQ5���ZMH������PMz�Q�����_�������s�UWe��
5�WEQ�,��e�=��#����O?=�~��+N}KgG������S��@�Ui�-����/�8�_~y�/_�$5�W��^
4('�|rN?��y��)�����g=�h���vm�,�Q����\r�%���K���&�-�z��*If����O?=��rJ�����&����G��YMz��`�Y�&?��O���0��R��fK�������s�i����N��I��J�����V����!Xc��M{{{��_=_z5l���r�)9�����7������6�vV���hH���kSE�������G��O?='�|r��(�vZ�<y��vm��j���5�+I��L����~9���s���f��wo���j�S;�kMFO�&�[C
�F�����������i��5�J����S;��?ij�&�[C
�.\�~��5�*�N������V������(���mcG�����q
��N
)�z��N:��]7��,�~-UH��<�=�X�/_�U�Ve���9rd&N��=����x����S;�����&�_eX�^{m.����p�
Y�p��~7n��~��y�{��#�8�q`e�,y$Y�����\�&���/�����|p�$�u��x(+~p�6����G�1�U�������O?=w�uW��,�-~�����.�,�]vY>����?��)S�j��~2�{��kp�7�!M���KY���Wuz��F^v��g��i���������HQ������������7��
������$��[.�:E�1r��n9�hn��s�]Zu��~���q���qcW���%X{��g^���f���<xp��Y�%K��������?�$]EX�>�l����>����'�����q��K6n�:E�0l�����n9�u��n9���!XK�.�����v�_�e����|�C�����e�}�y���<�H��������O{{{������7�7y�����.�4��@�6�v�?85a�{:-���{a��9Mv�V>�W��$�����g
�:5c>��nH@o���s�9'+W��*�:����_�"'N|��{��w����������x�������$Y�re>����k_�Z��@�����?69����g��/���kt�Q�x�5����$9���rM���,�����{W���	r���oU���M�81�]w]&N��u���^Z�����L�[�6~J5Y`gP��[n�%K�,IY�)�"��w^��]g�1"_��WR�e�d������[�3.���'���k��Vv
u/�z�������O<q��;��2t����������������WM�������J�E�I�&���e�����_&M�����^����MI���,�3�{����������������-g��i��v?�����{��1c�$eYf��)�r��{�������;tl�����y�k��T�vu/��k����W�X����w;t��~��,_�|��@w[xoRv�07�$c��.��^�5}��:4EQ�,�|����
���
6�c�X�<d����1����K,�S;�yU�o@5Y`gQ��~����NHY�)�"��sON8���^�z��Y�vm���w����N�E��N:)---��
I��M
��O�&�L�^��$���g���$IY���������&�����v��-�]�vm~��d�������E�$����>���g��*;��sk��)�����K&M��/|���G>��(R�e�|���q���?�i��������3x���Y�&K�.��w��[o�5k��MY�]�WEQ��_�b&M�����Q�O���]?��,�3iHV�|�����Ks���vY�e�5k���C���?lv_Y�I�U��$������~�1���������[M��45���~������1c��,�E�U��<Wv����^\�5v�������g�����Qm�`�����o[�vjhV���y��G���|'ox����������2����r�!�������������>j��qS��;��*.mmm�������}���5kr�m�e��Y�|yV�^�!C�d����8qb���7d���U��{vu�t~��xX�-*)�z�������:�Xxo�����_2������JS��'j�S;��7i�_M���4����������}m���g=��3�������3g�P6����`��RM�5���#�HQI�����9��#����n�-o}�[�$EQ����[2����d���k��V�vF
)�J��,�J���,�����gW���?��(�Sjj�E�U~��6�v:62��,�3jXV=(������O�&��Z�����Y�����Za���'�������u����#);;��e<��W%y��1�=��7-��s�?�h7���G�+�������G�Q]�JY&�������ugk��uH�����;V�Vu�i4U`[<��3�����$)�"���_��x%���=�W=G���[sw�1`���]�s�9[�����x��[}nY�Y�vm�������wY�bE��7��M��{�����1��LS��[�x���r�v�V����|&EQ����,�$�\r�v�Q�e�����i����g����-����[���jU��k����z(������LYrV�����ZZ2���3�����t��u[V#E�U�u�y�e����:[��l�������L���}�g�-�7���w����|�{�����??@w����,����3d��u�Q������9s���1?�tv�h�H��_Y�����k��v��eY��#�LQI�/��y��_���655e���5jT&N��u=_���y��d��j�=G�`~��[���^������m���y��jr=K�`���,u=@Y&msj��M�&��4���������\��YZ�6^�����������'#��&��(��.����OI���,@����h�S;��RM��ii�%�=�X���0aB��`�lX�<�P������<
)��4iR����s��HGGG���S�[�.�f�������������c�=r���f���U������s�s����_uy���!X�+�������O>�[n�%7�|sn����v�mY�jU���'f���;|��O?���>;?���f���~����>����r��'��};�mN�<z����j�=OC��GQ5s�%Z7�tS�;���|��ikk��}�]w]N9��,Y�d���~��9���r�i����.J�����
�������O�&�35����O��=k����O?�;��#+W�L�\������7���#n�[o�5?���r��7����;.���5�#F��^{����������������?���������%�a�P�I�=�k��&�35������v�-�2�������|&��~{����w���9��s�1a�2dHV�^�-g-_�<�~��k��&N�����9������x��|�s��\���O��|�k_�G>��n��-�iK�W����ZI��j�:�+)�"o��3{���~��)�2_����O|��\C��G��}�c���+2������������/����k�k��2k���x��]�WI��{�;���K
��9��,_����l��9�s��d��J�=T�/�z^KKK����9��CS�e���/����nx���?>s����+r����K_�R����_�8qb�����O���F��E]���_�A�_��_3s�����g��W���n���6-�?5yQo@�)�J�����s�9]�����6<��{�����?MM������.�������3g��������(r�Yg��}�{�KY�u��r6W��b��+I�<��:4eY��������_u�n�����f�����������%{��W��h�������l[�~m����5X��z]Vsss&M��5�r�-������W�����Y{�[��U{����G]�v��Wv[6�W�����|anjN��W]�g�uXI2`�����V�����;76l����k����n[���o|c�|�]wuW4�W�pN���>I���dz�^Y���c�u=���T����w�}5�����M�7�~����m���T���z]����<��S]��1c*L��x���y�=�����~�`���[�n�s���L������RM�gk�:��X�ti����)EQ�,�$�!�Rq����r�$�c�=�i���c�������$Iggg�.]��w���2=[gG�n�����jU�,�[�ts�=3�fm��%�x�s��*7l��,@��+
�V�X�+��"��sN����$EQd��)�w�}+N��V�^]3<x��E�����Z�B���gn���z*O?��6�y���w�^`��wur�7����u�����>�/���gq��
�R�u��Gn�����,^�8�=�X��LY�)����_���9i�6-�8p�6�Q��o��9���w��~�]�\s^�~M�I�3b��U�z��`]w�u)���A)�������$9���s�q�uK��d��u5s�����������;�	�������$�u�U�vV��������5�kG<_�U�e����������9���+NV����_��g<���[<�9���:�s��
��g�c��w����?��:��?�1V�U��6�8p`�
��'����q����;.���uH�32�f^�n�6������3��?��?��SN��=?�pN:����:����{g2�o��g��>�e���59�����w(GK�2�[��d���$M��i��>�YR������kv
��U�Y�f���eY��1c�d��1;|P������M884r���[�K�����3l��uH����Pk���'�xb��/^�8]sSSSF�Vf;����������RI�W����w�}k��{l��o����3p����l�����%Ct�=��f���������m��}�m�<`��6�v?��[CV3e������k�?~.\���o�����u�{]wEz��9�������
�z��C�f���5k���o�joY����kj��?��n��L��$��]���Z��������Y�r��wf��%Y�bE�}��m:�(���w��S���p�	5���~��9���^q���^�y��u�c�����Z��@��pn��2 �u�j�l��`���?�����r�-;tNY�)���R�,�����O��Y�$��������>Gy���)�2g�}v�����������Y������w{M�\Y�!�+kX;���s�g���O�-����,���e���9��~g3f�������k��8���������������������c�X�2=������)���Z-�����p����%I��H��B������+�o��n�)���/Y����k�u����k������������w|����_�E�%I����3f�����������O<��}�s���j������Q�����>���{k��O�&��jH����|�[��*m���>�����SO��I�2p���w�\sM���7e��e�;wn~����{��^/^��(������K/�k_��FD�����=Y�`�+~�x��s�1�}w����?����5jT.�����mo��u��$,��'��#Fd�����+��c�e���5{O<��|����_�k-��l���O��55�����I��,����_���9��s��W�:���������n�����:*��{n,X�U�t�}�e�������R3g����g��Q5�+V���w��y����������r���w�;��9�����A#�����^��f����w�KQ)�"�{��r��Go���������|���NY�Y�jU���wd���uJ�sy�����{����?�
z��:�����g.���0��	�*-��k��jrl��z_��?�17nL�E���������!����s�Wd������>�/����n����7���c�����v�;����5+��w_V�X����g��w����}��������m���S���-�^��`�������g���[���g����O~�����+R�e.������8p`�d��Z[[s�QG�����:
��]��x�vm���+�Gh����-K�E�	&l����z���[���<���;���Y�fuSR���mn���5�ur5Y�E��:::����o���,�$O=��+���{t=?��;��wZ8�v��5IS����I�^�5|������Wo��#Ft=/X�������z^�b�vg���6)�?�����X&LH��e�e��m����������o��y7n�C=��(�$�������d�}�k��V�`[��k��)]�mmmY�z�K�y��^�����o�1mmm/{��~���^�:eY&I�����z�%'�������oz��`M�4)���[�|�m�����N:)IRE:::����?/���G��>��E����7���C�pmsk��{$�FT`���+I�<����_��W/y?m��p�]��W^�������7�����:W^ye>��O�����E�R�e�����33a��F�z��{j��l�;������|������$���+��/~1EQ�|��o~3GqD��,��s�=���?\����WI��_�|��_�{v���mn�<nJ59�GC
�����������>�$�5kV���7������:�:����m�'v�n�R�@�2Evq������3�x1#���0��:��qA���:�0����NK[V��B�&)M��i��9�>��YN�}N�z]W��|���s�����y'q�I'�������k��,��f���g2��d2��fc��Aq��w������9���O�+ON�m����V{����z��XzY����lKK��tW^
��/���>�������C����.^}���k,���?�������g�Q�������O���!9�vY���f~tDG��
��R����vZ�������o�/����Nd�����c���1}���#t[�o6FG��>�v��g"�����	rr��)��?�1cF��1#�9����>�����qh��9�V���Qy�U9���l@�N]Eb>x�����tz���)���I[������NDDf��(?�����k��,�>�u}D��a��I���>qmJ��U�v����*9��h�QmK�	P`����*��G&:�	P`����
�*����@(��C;�"�y=�V��`��������ZE���(
��P���<�yIn��N����W'���y�( 
��P}Ur�P����45F4��\�T����4T%���M1�yq:a
�,�>R�SVE���l:a
�,�>R_��+�����(����k�&�*`D�,�>����m������i�((
��@}Ur��%���N���4�T�������(D
��@}urV��.X9����-����&oH'@R��c�U��|dD�����(@
�r�a��	3#2�t�"X9V�s���(T
�r�m{��7�k`$�������c����u��^]�#GG���.dw��O��7��2!@���j������N���D����#��b������^�������,�M����@���v�^ZY�'�(
��o��	}r����1}�?�����|�O�	P�������7��X{��;c����u���c���<���2�(?��(?��^]��)��w�FtdK�]�v�ac����mz��������*9�l��A�m�� `ZCur�h��N�D0`e��U��JX}N0`5��u]r�b��t� 
���nQr��6����N�D0`5T'���"�N�E0`�W%���y�``R����7�k�Ms�	0�(��w^����'�1�iajyX��T�(9����(���N�F0 �W'���y��`N6�P�\�l��N�H0�l��h��\�h��N�H0�4T'���[cX��t�@
���nQr�����`@R�8
��y�{��	0@)���m�o%��U*��'X���zIDv��sIi������`J}UrwpDYY{:a(X���s��#��0�)��l6�aqrm��t�d
��c���m�k�H%
��V�v�}Yt�sQ�tG�����:�v�'"���K6D��>�K�2!��(h���\<�����G�y"���I�\�=+I;��,���>�ve������g�@P�����������[���5w���Q^�'�����'�"�w�J����q�:�W�2�9W���&��)+�a���O=�W�`�`��*9n]���j���O�����5�����I�NX�Ms#�I'��(X�`�K'9�(H��k�%�*6+��`i���l��s��-Fo})�@��,� �/J�����-�� �`��:9W6�M'9�(8�lD�NXM��	@�)�
���-��k��`�
���S�(9�^��+�	@�)�
N}Ur�l��t��`��:9W4�M'}BPP�o�X�<�V�4/�0�	X@AiX�w��h�Q[^N-��((�U��r��Q�mM'}BPPv*�[^�N��,�`d;"'��U���(�
�����[�kc�`�7
���Q_��G��V�1�,�X@�h�J��H'}KP0����DX��,� l���6���R������lp���d�o��(^�l�[?X��l�l6��k5n��;���o����-�����X@����'���O��k�X|�}����k������������+7���zf��>�'��������o
���,�q��k�?��>�'�S�v�85,N�%�cB��(>�W�V���]�:�����
���X?��^]��)�z��:9���b|�������\�/sp

UI����P��+���������
��5Xt�,����F�6'�*`�E
��nk�J�C[j���!�0X@��W'���y��()���~Qr�T�@7(��e���
+�kMs�	@QR�tKCur.������N��,�[�����-/FI�=�0%X@��\�U�4/� -X@�u�G�^�\�l��N���u�-�D&�����/�<��~h����m����y���h)����*9�XC��	@�R�t��X���>� ����5_����}�{_���8qb�a�j���C���H��5+N=���c0�5o��X�\��(�0������:91*[�N��,�K�%���(�����)����:9O<"�?X�>�h�X�$�6A=����5���kf����W�v�n����l��X�n]4(��'N�a����~��*9WN�V�J�XE�3��L,[�,ZZZ�eee1k��8��s��k��q�������m��x���h^��X_������s;�����y�c��<'�?P�U$/^�������?~��??�����������o��������5�����sjjjr�����ns��������<v����s��C*�#X���m��+_�J<�����_�*F���k����[o�5g���,�����tV���h����LII.#�Oi�)`�L&f����v[�����U�VEsss���D]]]��W�����*����=��Sq����;RJN1�_�����[^��������D����8�QY�����:+>��O�!����'N�'N�s�=7����).���x���;_�������v|����Wd���5����+bpYsbmT��qH��(?���zieeT^sMy�{�<'�/��f�i���ZZZ���O�y��u��?>�/_��
�����Y����:���&�?������*f����,�^�����Hr�����Y/��N 
^uuuq��swz���*�U^^��{ov�a���(�z���%T=5~��?~|��CajX��K:Z�"�4�0�{%i w>���������c�=�R�I}ur���(��t���)��g�8����t����PL��sE��t�0 (��g�L���SJB����P�\��<7�0
���A�%������P,��F�6'�*������AV?�z���<n����P,������(o[��� `�3�=�\b�2eJJI(�����i^:A0`�#7n�����I��q�)��X�W%�����`�P���\���q���y���q�9�����mS������y��`�P�U�����9^z��.����]w]��?H�_}��1a��\��i�N�%;�c����	����=��#q�����'���o�UUU�����q�6m����'q�q��7����k��O�/����L���J����%�]�o
r�,�����sc���1d���<yr�=:JKKc��uQ[[�������o~��3fL�#Sd��se��t�0�(�*��o���zk��}�#�9s��������b����8�V�4/�0(
�
�?��?�a��>�l�����c���?b��8��s��������O�SJ����m��k
��X��3��3�<3""���c���Q[[


�e��������������?<�<��(--M95���*9��9��5��EV�6lX{��q�����~�a��������$�@:����,�E@�"6�J�)� _`�T_��
�xO��t�0�(����*9�XDII6�08
�`���k��tr00)��fG{����k�H'SY�����ac�������u���rpb����h]�,���`A?�����S����G�������;:6O� �$�@�e;"�{����t�>���</��]
��Z�v�����t�~�����#��a"xWY������i!�������c���b���Gf����Lii�=;��~{�0)��~��*9����������Y��duDD|����S�v �v.���<7� �
���i���6�V�4/�,�;
���iX��K:Zb��W�	�������<z�KQ�mK'��,�g��sE��t��(��~�cGDCur�r��t��(��~d]mDksr��I�E�#
U�yhKm���N'��,�G���sE��t��^(��~��*9W6�K'��,�'�m���"�V��������\��9Fm}-�0�
�����J��/EI�=�0�
���h�N��[^H'��,�:�#'�*��O'��,��.�h��\�����������\1)bH{c:a``A?��S��#��]������<qf:9�+`A�k��qUrm��d��P�E��:91��t�@W(��"�P���?,��,�,�jr��l_�5���ku���A�������
-�l��@�(��"��qK����E}���-��c;n�x����)t�,(�o~%��O�(����M/�} ��.��i[Y3�K��nZ�����B&�<0���g`A��vD������a�jb������)/�q���ec�V�}�,����_��cXb���������a�#���.��VVFIyy�f��P���*9�o_��h����K'�RI�������in:A G`A�k���b��t�@�(������6�V�����
�������%Fm}%�0�#
����U%��[^��l[:a G`Ak����i^:A �`A��vD4,N�UnV�@�S�j]mD���ZE�,��,(P�������Q��:�0�C
��@�W'���y��S���*9W*���P�h���
+�k
��'`Aj�N��%�1j�k���S���*9��2J�����S���:9��v:A�(������8�6vDm*Y�/(���vyD������o���,(0�U��bRD��-���>�
L�NX�H'�XP`�����������
H������k�H'�XP@�������J'�����@��w_�y?,�����zGIyD���y��g�f��c��|G�>���us[<>gJl��>��X�L�X�.� J���'�T~1f���]/��\F��R�y��fH�����b�����^6qbT\ye�c@�����7�����b������w��q����;>"�J[2dHy���dH�J��)��<�vD�~+Y������K��
��$�0��;�u[ib��i^Ji 
� ����s���1��.�0�X�G���se��t�@�`A�W'�����	)R�y��9b}mr��I�,��������%Fm}%�0�"X�'���y���$��NH�,���E���i^:A e
� :vD�^�\�����I��������ZE�,&X�
U�yh��(o[�NH�,�����\�4/� P`A�/J��
��`A��)b���Z�,0X����s������t�@P�}��*9���=J�������XCur�zK:A�@(��>����8�6�`Xl
���]��-���,8X�����s���a�v�
�,�C
;`M8"�PH`A���k��tr@!)K;��mm���>�����l�������#���X���y��F��s�t����?�=W�cKz���=v�'���@�{���]:��i^����*3|xd�t�0p(��^��%bs���>0�3V����*��o#SZ��\P�����aI����e:Zc�{�����\+)��I�b�WD����rH���
�,���E�y������7��8)�@PJ����*9W4�K'X�C�������JX�O
������}Kr�B��,�����\�}Em�K'X�C
;`U6�K'X�C����b�,�
X�-�#��&�*�`@W(��h�N�%-1j�+���"S�v��������U����5*++��C���gGyyy�����
�Foy1J�m���"��H<������|%^~����>b���������o��c��9��S_��+����PI�������'?������X~�e�������8�����g��c���cG�������
���`������������K������i��������G'^kll�s�9'��S��W�-�hmN�U4y���`��}�k��_�"�v��W��+b��e��+��������8���������c��M��< �W%�a��Q��:�0P�`�u���m���X�����;��NL�8�s���$.����;wnL�:�s}��U��o|#_q����<�dq:A�H)�*P��/�MMM���'�7�x���4iR�u�]��o~���n��>�8P�W%�1%K�	EJV����9s�$�n����d2{=��3���N:�snjj�����O2T�#6�L��-]�J(V
�
���s����s>�����SO���W^yeb~��s�����\6$btfY:a�H)�*@=�Pb>��3#��t��3�<31?��S�u���e������E�dv��������&���gw���'���S;����X�xq��Q_��'�L'3Xh��%�������;������������&�N(f
�
��m�b����)S�t�;�t��^�"�qYD{Krm��t�@1S�U`��]�l�s4hP�?�[��4iRb^�fMN�
t
U��bR���t�@1+K;I[�lI���
�L&��k>|����5k�Dccc�������}�����w���k��;��M�G�����E����Xf������n_c���{�fO|����[o����)tk6G��v�����(6���>N�OI�HjiiI����5����m���L�^e�������$P\`����������kl��}�����nx,Fn}m���Q�b�Yg���GY�H1bDbnii��5�m���k��5�\����uNMMM�����w>�}�����S{=f���8`��|�
���Q�b�G?��M���P�`�������#��F&���5�n���k�����c�����N��t���t��.��20�]%i i���������X�fM��QWW��BqP<`��C���X[�bE������zh�s�JY�����o��v��x��8����|��%Kv�^�o���kjjR�����B;����t��G���>�9��;7.���.����������A�����u�.Y�reb>���S����+W�1���cK�8=p���&��<��l��}����i��#F��Y6��R�U�f��c�����-[O=�T����~���;��\F��L6����]}���������)��O>�dd2�=�����.>��u�#G��e��%���i������Ow�S�L�!C����;jjj������|��8����@���?{:��gO�?��?{:��gOP������o�+W���O9��������e}��^�������nl��%""�~�����;�_��n���������N�]{����_EDTTT�y�����s�����3g��^��(~�t������(~�t���������s�1=:�$�9���c��������������k����s���#|���={v���v�O�81����|��2X��o�s�=7����|'8���>}zs�11f�����b����:4��������sb�}S�U�JJJ�g?�Y\|����;v��e���W^��7&^3fL<���q��'�1)@�)�*p�������$~�����G�����\sM,^�8N=���������5����c�X����������.Z[[���";��8�����<������|��q�����J������R�@^)� �`�Wei�B3n������3�������?{:��������?{:��gO���f���C0p����Ey���R�@^)� �`�W
��+X��,�Jy���R�@^)� ���@���[o��b��U������q����������<'�hii��s�����6l���������@t�A9�Go��}�(��(>����\�2^|��hhh��7��A����"f���f���#G��>=QWW�-������qc���FeeeL�:5>���<�=P��n�����7�x#6l��G��i���GS�LI9a���r!��Fmmm,Z�(V�Z7n�!C�Deee��1#�;���>k,�����K/ECCC���#��GqD|�������CSSS<�����o����c���q������c���9�������������QWWMMM����F��1c��QG�vX�����>�����_=V�Xk����[�FiiiTTT��I�b��Y1a���T=����-���z+6l��l6*++c���q�	'�{��������=���UUU�n��(--�	&��Y�b���9�}��W�X�>��^���^�eOG�(�"�|����W�/���n_1bD\~��q��7���c{t�������[��{���[����Y�f�M7���w^���[�x"����O_�eZZZ�{��^|�����_}�������?�����k������}�k�����_�:~�����~��X�j��---�SN9%���������^��������X�`A��??,X/��b455u�~��Fmmm�������7���x������u��M�4)�>������������{��K.���������/�t@om��!|��x��G��'���k����A������������)����������i�z�5�������k��9q�w�o���c�������_�B>���Z�|y|��_�������L&��rJ�z��q��'��>@�����<��x������_�����?z�����K��k��C=�K�X�vm�{�����������~����93��������b��a]�Oo������>=�P��w�����=��d������~��q�����t��-��������|'��_��c������7��_~yd2�� W�[[[��W_�|��`��x��7����3'.�����=�^�Bt�-��������/������{�uN1�c|�(&�����k�=������S���o�W^ye�w�}]:~��q���������SO��?����������K����~<�[���|���^�'{�W_}5.����~�;�\rI�u�]}�����~{�v�m{��{��8����������r�������������o�*���iS\s�5����[�}�;����������_��W�}���������|�3q�]w��LtO.����������Q��t|1`m��1.�����o���:����/3g�����������+���y��f2�������oW��b���QWW���
_������o������3�<�G��M�?�����N���]q�=���>����aC��=��������3ft��E��y����/���g�}v���w�=�����3����>�y��x������`�`y�
�|�%�>��^�b��=���&������P�:::���.��������i��������D��s��y��u�>�=�\|�#��?�����?�N�������������K��K��z"
���O>�2�-��O?}�x��q���������~Y�'?�I��_�E���/���n?�d21a��8�������n���#�<������[���ri���������,�������={�n����o�8���������>8������+6m�����sv={: �������JKKc���1k��8���v���{��7�<����eK>����m��������j���q�!���G��O��l��8�������[����~�\r�.�W����c�9&&O��(��f�q�w��>��n�T�@U^^�rHw�q1k��8��w)�jkk�[o�5�������;vl��93N8��8������r�c�/_g�uV<��c=���TUU���j���q��G��Y�b��	���p���={v,Y��[�\�ti�~����_�1"�:���1cF4(����>��sN���t�^�z�x�]w�����w���g�P���{�;�{M��#-
����}�k��_�"�v��W��+b��e��+��������8���������c��M����
���.�m��u�x������������_����Gmmm\u�U�sx����7����r���>D�{�������#>��O%~����4����X�jU���������W_�
6�O<�|�O<�D|�k_��O�wC��K/�4���'���>^y��x���c��5�p��]�j�[o���w^tttt��t@>�1"����ys|�����w�UVV�w���-���W�k��,�7�|3�n�/��B|��_�����4Kw}���������]
����q�5��C=6l��+W��/����Z�[�.�|��8����,X� .��������������������b���sIII�t�M�z��X�ti����������9s����q����;�t���z+������z�{����'��5k��K/�+W��%K��G?������������O
�G'N����������������uk,]�4,X/��b�����u���;����''�����c��9��G&��|�q�M7�c�=�����������y���k�����������g?��������'>������
�.� ��������w�y'~�����/�����d�����+��]�6>���RP�'����������=���0��_���Z����z������()y��f���7�pCn~X`���3��)++�!C�������+PL����w��hW?3*�}����E_��|�5���Te���]�9rd6":��~��{<~��U��S�&��������|��_L�3m��l]]��������=:�~����]���!�-��(>������?O�����{=���5{��g'�9rd����G?��|�c��7.�������7v���~����Lw�}w����r����f�~��SO�~������g?����f�|���^�����������N9�����k�tnGGGv��
��O=����L&���lIII�_��_z�����2k�����S�w�uW���y�����g��o�v����x��}��|���9�]vY~��Z�dI���4���?�������VTT��s�K.�$q�q����i�n������}�>}z����G?'�?���k����.�~���1���cL�0!�c��=�����]�jU�r=��c���'�s�
7t�]u�u�eG��������W���9��{o�gw��}��}�{��*++����{<����K_VV�}��7�t/�\>s=zt6��dg����?���d���oe�����m����rJ��s������'<{
��7���/<���}r�B����+P�������${:���^����A:�����<�?��.�1�����Y�1bD���|�������|���s���/��g��|��lq�@���^�S��T���|�#]��b��lYYY��G}�[?_W<��c����n�w�$���={����}���&[]]��/����O���[�n�E��hnn�N�>�3���^�������_�������w����������[|����y�P�u��&2~�S���9w�u�.{������SUU�-))�<g���������m��eg������w����`g�/�����g���}���/$�q�A���l6����f�}�?��?$�M�4i��l��=;e���y?���y�'?��n��r��������*�,�^�b����b����+P�������w����������9s�$�n����d2{=��3���N:�snjj���������?�-[�t�'�|r�q�{�G&���o�9�v��wG6���y=���!������k/�t������~�K��L��|b����K�v��g�#F���y_���/��7o��9�t@_�>}z~��QR���~�s��������]w���
��{��M7�o��VDDp����~�W���r�����c����:���4n�������>��X���aC<���s&��[n�e��]q�q��v�o��v<���{=������������/��;l�����������]w���|{s�a���Y�kK�,��}>��O&�e������s~�~��1v��n�w��7&�S���EUU�^�y��Gc�������S��+����v~&����,6m��������g�'�|rTTT� Y~y�
������W�}���w���6X����s����s>�����SO���W^yeb~���x�/~�����'��vZL�6�s^�zu���]:�;��>D�{�|�e��_���L����p@b��qc���k�{l:�s����U�V��{:�X-\�0^y����������:*�D]�p�������9���g��){:���/GD�[�.���SJ�=�P���w���zjt�A�<���d���}={��/���������b�����������K���������k���C����/9�����+�z����]q���_���SN9�snkk��~�i���/�9#�^�U���{�7�{}�=iS�E�=��C���3���/����?��SO���[w9n��-��3�$��:��.�#����>�������.���x"����O��2�G�N���m�r�����_�+�Lf��mo��(f?���'?����t][[[\y���c�������?��{n��iO����]����T1�������"v��ro���K�&������c���]����f��]rtWKKKb�����{��}zc�=������?[_<s,F�����1��
�o����tX����������'N��S�v�����x��]���������y��i����w�>'�xbb�9��jkk#��$�������|��/��!_{���>:1/\��K��f��{��������tO=�T���]mmm���1f��=oO��!��
�'z�9���~���h�����������������BQWW����>�����zj�=������Y�f��!C:����]>���}�?��(++������\��3�Y�f��>���O��zh6l��M�:5�����{r���v���m��;�����;�!C��1���{��]�����W�����w�����^����B��[�dIb>����u����|�|����������k�q�E%�����q��}^���������s>��cv�P!Ms�����v���8`�����j����j���y����d��-1g��8���b���1d��7n\y��q��W�C=�l6��/^��v[�|�wt�!�������g�M�x`<�G�Z�re,\�0�}������cqT_jkk�����Zw�ZC�����'�<{���w��x&z���v�������#n�����e�]��{������AFD��1c����;������yOWSS���]>���c�����o�K�,���{.���555�����k�>��^��(�{��{�#{:
�,zd��m�b�����)S�u���_�t�.�����{���������k�M������{�|�e�<��8���:�5k��_��_�;����k�����O�����A��?��?�����}������O����=oO���:�A�?�x�����������������oGkkk�]�6����{��^�{��q�q��������#����hmm����N:)��o�&'���
���M}�#��5{���8qbp�q�����'�GqD�?>�M�W\qE��7/W��j��e����C����c�u������>����0��������$��?�#2�L���q����'>O?�t�����^{m�����sO�H����i�����������x������/��5���/�9�^���3��LTTT���'�tR�p�	1c��=zt�p�	q���v���>��^��(�{��{�#{:
AY�(Nk��M����A�b�������I���5kv9f����'w����_���u��tGGG�[�n�{�T������{�|�e""����8������*""�{���1cF\|��q��'�������5jjj����e<��3��6,~���	'���|}����}�w�}���/�|��������&1�5*~�����^��S��K/���rJ���?�.���bv����x��""b���q��w��K��t@!x�����E�����ihh��k���q�=��=����~z��3g���v�g�d��g�;����8o��F��fmmm�a������_���x������9�g����<�������9777GCCC���������i����9��x��Gb���=��r���!���o%����kD���c��e�k��1������|�XL<{��?��O�������c���q�w���_7�|s�����z�����W�?�����|���������e����}WU�����R ������ �4YG@Q@���/��
2��,E���"
���J/a)!�FB ������\97�-���K��g��s>�}8����s>�c���;��`����,�����!???���[��_�����������$���j��3f�f�����L���k����;wn�s�+���{��>(T/gddh����{��9��qc�����<j:���������k�����d*U����{N
6T@@���=�U�Vi����9�n����m�65i����5))I����y<f���]�i���Zjj��
f����S��5+��6lP�F��b�
�i��@r�o�����Z{�<�����������C���3gj���V����P��]����A�����GC�����W^���cU�D�|�)&�IC����W����0�9��<W����)�5Gw��+(Jn���>�@?���~��G���
{��WPT9R�ec�+5
`!_,�/^��sX~A��X��c�/�j��:[������\��������3fh��z�������m��9r��������^���{�cooo��;W���V�Q�we������<x�>��S����>|�6n����{��gff��g�UBB����~<��s�{�����~��$�v��;vl���j:� eee�������g���� ��1���T�TIQQQ��������
*(  @W�\QRR�6n��Y�f�����9��������o�n�&�v��=�k��y��I,�}���k�����W�����5d���_%''�E��'O��5k��3g�|��A�w�����]��WPTyxx�E����T�f�T�N�*UJ����t������U�V���?��[������5`��\�R^^^���}���EMrS�5����JM����@���/I>>>������7o>�<����+����Q_$%%�g��j�������h�"=���������������Ks��1�&M������KM�U^�QQQ�?~��W���o��+WL�?�e���u��7Oqqq����5g��|�\�P��i��Q�������g�r��v�

�?��S�Ni�������j����� y{{�t��j���F��c��i����Z.--M�=�\����Nk��5�����l�2�n�Zm����'
$���W5n�8U�^]����233$��V�\�w�}�>|����m��9((�Xst��w��K�������[5v�Xu��Iaaa�������*V�����k��Y:~��Z�je��z�j��93���:���^���������Wj:<x4�B�Xv���1V���<����+���W�7nT��
�r�J���;j��e:}��222t�����OS�LQXX�y��5k��iS?~��kt�+V��7�0�������G�5�����jooo}�������:�M�6z���
1��gHIIQLL�y<t�P=���N�CM�3f�����!6z�h�����s���(**���$���K&L��s���������i/wZ{��c-(�>��c�L&�?7n���3g�j�*�������3���?�����]�����f�������4:tHs��5������)S�(222�K���c�
8PYYY���O>��?����<��Uk��k����l�R�j����J�*)..N-Z�0�'M��7n�:���1����H.P�tM��}��t(h��|)Q��a���a,;�Y���y����W��p?��/�=���(]�zU������s�*..N}��U���������@5h�@�G�VBB��v�j>GJJ����������z��g
/`w��E_|����������������*W�l���_~�0��sg�E�9r����$I����G}���g����_~�����k�������_W��m
�E�9=�;����u:j:`???U�TI�������t��5l������4������+?<<<���
:T[�l����
_�����[o�u��s��aEFF^6o���~���_���@Ap��ca��+(j�/�/��B�������n��\�/�u�;��:��Gk:�}����P��b������n���~���s���c��dr�����������������
�������j��������o�����cG���3�Fg��k����
]�[�h����N>>>v������j�&�<������3�o����:��$i���Z�b�y<}�t;�������j�*
4����w�������~A�~Y6G��a�������[g�6���^-������u _j�����XC��s��i���N�����E���!���[G�uj{$%%�K�.JMM5�y��]�VAAAv����#(\��X���
���5k*::���Va�c��
�*Gj:�}����P��jx������p��C�8w��a\�l��X���=�P����������BCC:�5��������U���o���h��Qv����_����!6w�\���~>|X]�vUzz�9��A�Y�F�����r������U������
7��=�������T�~��vnK�t��6n���}�����;k������r�ut����,1==])))N�aYgY>{����^-��)S����


���
��:=O�����M��d2��/�pzk��?�N�:�����X�����3Hk�����k�E�+�3�^@Q��cG�8�����a�+(�������'j:4�B�����J�*�����:����k��q��]���j��N������T��p?��e��_ow��A���v����4,��8q�����������;���K�X�Z��n��|}�����:u���:t��/_�|_�t���4���^�Z6�i�����N��q��}�r�����s�NEGG���[�X��-�b�
����
		1�������Q��������7o:��A�����^�z�2���?^�N�rz���{���msz��\�xQ�:u���'���e�*...���=����p�������Q�5���p��ca��+(�*W�l��^Y����������}���Ca@,�����&$$84���#V�����U����W�III�q�����cs���D
��3g��cG�C�*U�(...�_�����n��9b����d�Z*L���+8p@��u��k���F����l�R�X1�����N?xx�!�H����ah� ��
�Gpp�J�*e����oN�cYoD��\�rEO=����
��u�T�V�|���KLLTff���-k���pCCV<������5Qw�c��
�*{k:������
��o
64��\JJ������b�����.""��Lrr�C]�n�j[^�3��>H�q/��qE-c�!?/
[>t�{����p����S'�W���/�������j:��J�.���p����C��p�B�s�#j:P��=���;�����X�:u�v�Z=�k�s���kq�T�L�������w�6<��P�B�/�X����_u���sQ�W�\#u���_Wdd����c��(QB?���4h����/_^���7�322�{�n��S�<x������a|^k��P���U��t�{�5
>W�|�����L�b����d2������u������W�%rW�dI�i�F���7�bcc������a2�g�EEE���(W��=�p?��e,�?��k���p���l�/_V���u��& �iIDAT1s�T�R���U��5����t��EGG��O>1�w��mW#��Y���g
�Z�j9��V�\���Y���~������������
��V�Q���t��)u����<�z�����-�gb������ �����l�Y�w������<�����1c��k[��j�����p%&&J�_S�m���M�66�\�~]��o7�=<<��{w�� /���JMM5���+��<��]P9������={^x.^��V�\�'�x����y�������j���]s�@���5����WP�m���0����;�1�{E��5�^�DM���X���-[*44T/^�$�<yR���j�������t"I=z������h�/�y�����r���JJJ2���+�������(W�����~\Q�T�V�0����{�D�6m�d�\������0��:���k����8`�j����W��SrP�w��W/C����^��M�������|��a\�N�n�k����s,��R�xqu������t� ����c�����aaaZ�~}�=�����-Z�����y�~�iy{{����������Q���y&�I.4��Y{�6m�y<o�<�`}����v��y��iSU�X��<kV�^-��d�)SF*Tpz���0�_���sd�s�����gx��X�bZ�|�:t�������Zu��z���l�E'&&j��M��z����rM�}�z���b�Uiii���o
��;�y|a�c��
�"Gj:��Q��A��3	�����l�M�8��"Nn��_����<.Y��������P@@�y�y�fm���j����'bC��k3��\u��/��qE-c������Z�l�]�g2�4y�dC�u������k�#n�����h���������U���iS������m��z��G����O���?�9�����?�a��zV�Q�gKMMU������h��)SF����^��������h�"C�g����T�R�s�L&M�0������+99�<�Z���
v/����e����JG��:���[9�W����6����7oj����X������h��
Z�v�!fk�Z~eeei�������1OOO-^�X��wwZ���zJ�*U2�����`���&L�`X��g�����X{�ELL�����cu��-��{��WP9R��������������*Q��y�i�&M�2%����;��C�b����BCC��S�lY���5�������9��?�������� �5*���%''�������/J���Ar��EGA�2���z��'�#F�M��d2i��1��q�!>h� ��$�����������������G+V���O>i3�#���������I��7�|S{���s���w���/���W_}�j��>�sj:�L�������>l�k��u�S�������j�������a����[w��5�*T��������������
/�,Z�HK�.��������b����|||���W���E���L
4HW�^��x���7�xC��7�j���^x�jPt�=Z����CsRSS�c���c^^^z��7s=����;v�RSS��y�f=��3�X����Z�V����[�p��9#G���%K�c}��gNo�����q��b111JHH�s��_~�����^^^9^�x���
����'k���v�����[�7o�!>|�pU�P!�y�{�5p6W�t�{5�����a������?���c
�#F��w�U��%���n?���^�u�>}�|\��u��a[����������o�X��U5c�EEE�_�>{��&M����g���Gv?���rRR��U�fs�+����{����e6m����������w�yG/�����/o�geei������r|������}�lv���L�q�F�k�.�����E��#G�g��V��&""��B�DM
���[u��������+W����^+VT��u��i���6m�d������!Ch����K�F���(3{�l����Vs��s:{����}���q��Uz���8K���s�L����Z�h����4i����\�p�B
2Daaa���������qc��{��v����3gj������2����S�|��z��e�Z,�-I6�&�m��a�3g�!��q����o��|�o���%K����M�/_6[�~}���[���6��8qB
4��7��
���?6<O<v���������0��e����]&��k�������f���������"""T�X1�q&�IG�����5c�]�x�����M�:5����S�F�������(���G��7W���s{��Mm��U����W_}e��|}}�u�V5i�����Z�j:u��y�`�
<8��'N��	&b}����a�l��T�F
��Q��1�o�V�F�
eK�*�i��i�����055U��M����wC���+���O?u����[sLII1�0�z����F��.]��zl���U�x�<���
�����Z�l�~���c���]�v���+W�h��5�����o�>������s�N�.]�j.�����8��j:��Q��A��[VV�z���U�V�^^^�Z�����������4������V�Z��g���z���t��-C<88X��WWZZ�N�>m���$���C+V������� �U�Ar��E�+j�)S���w���g��US��e������d]�r%�1e����;l��,9� �����a���l�t���Q�4(������j�JIII����������g�=����K�Fyq���%j:���	�=�n�e),,L�J�R@@��^����O���k�^���������u-�����j���v��e�����z��������'s\ghh��n��Z�j��G����+
80���)SFU�T��t���?��W5c�����_v�{���(,,L������Qzz���9����\�1h� ��??���`Y
		Q��e�;w�(--M�N���\J�_��+V�[�nv��m�e�@�~�?>G3��9rD�[�Vjj�!^�D	�������JJJ����
?o������������E���9����Q��QY{EMn��|}}U�R%���K�.]Rrr��Qz����k���z��G�����?Q�gqeM��W#j:<(���v�������5`�C����:y�������~K�.�5k�8�?�m������U�T)C<--M{��URRR�_���_��_���R�������y���5k�,�����Yrr�~�����?��M�4��-[�z�.�����P��6o���M�����:r��v���������&L�`W�+wBM�������c�%$$����B�
Z�v�����������]�:����:z��8��:�U��
68��J��%K��hz��h���:s�L��W111�>}�Cy@�������$���W;w�TBBB���5s�L-X���W�ss��e=zT���������'O����i����c�����E�:u�a�U�Z��v������c���h~��S'�]���W�k�� edd(11Q{��������'O��(���������n~%�:�}��aQP5�^������NQ�xq-]�T�|��6l��qz��W���`�a^:t�����1"��@�5j�H�~���,Y"___�������T��p?��e�
�������L�2V����P�f��`�m�����k�����J�*i������O��q����������������
]����:h���j���J�,i�xOOO5n�X�f���'��sg\��J�*���X��3G5k��z���cu��A=��c���������C8p��+��qm��Q||��N���: ��K�j��)����m�������k���:q��F�a�����������k�_�����l����S�=����k��V���Y�
t��A�3F!!!y��#�h���Z�n����]w����W��������a���%J�o����i�V�^��e�:����1�{���5�^���� x�,?U
8��'�s�N�;wN���
V�:u��U+/^�)9n���m�����#JKK���������ys�/@��+��������eL&��?��{������r���+���`U�RE��5��b������C�i���:�������U��u��V<6�t�]�L&%&&���:s�����t��-($$D�+WV�f��j��*��={�����w��t���W���7on�i���^��-[�����JOOW���U�J�j�JaaaN�nYYY:~��N�8���O�����}��J�,��� U�VM�7��z����JHHPbb�.\��k�����SAAA
		QDD����k�K����ok���:t��.]�$///U�PA�7�w�T�k��n�����%''+%%E��]SVV������u����s��3w�c��
���k:��Q��Uh���|����Xp)`��h���\�Xp)`��h���\�Xp)`��h���\�Xp)`��h���\�Xp)`��h���H�u�:��pIEND�B`�
#8wenhui qiu
qiuwenhuifx@gmail.com
In reply to: Melanie Plageman (#7)
Re: Trigger more frequent autovacuums of heavy insert tables

Hi Melanie Plageman
Thank you for working on this ,Actually, there were two patches aimed at
optimizing vacuum-triggered processes, and one of them reached a consensus
and has been committed:https://commitfest.postgresql.org/52/5046/ ,
https://commitfest.postgresql.org/51/5395/, Maybe referring to the already
committed patch and setting a maximum value for vacuum_max_ins_threshold
would be more acceptable.

Thanks

On Thu, Feb 6, 2025 at 6:08 AM Melanie Plageman <melanieplageman@gmail.com>
wrote:

Show quoted text

On Thu, Jan 16, 2025 at 5:50 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Thu, Jan 16, 2025 at 4:43 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Fri, Oct 25, 2024 at 11:14 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I've done something similar to this in attached v2.

This needed a rebase. See attached v4.

Whoops -- docs didn't build. Attached v5.

Outside of the positive performance impact of vacuuming pages before
they go cold (detailed in my first email [1]), there is also a
substantial positive effect with this patch for large tables with
substantial cold regions: fewer anti-wraparound vacuums and more
frequent normal/aggressive vacuums

With the default vacuum settings, you often see an append-only table
devolve to _only_ anti-wraparound vacuums after the first aggressive
vacuum. I ran an insert-only workload for an hour (with 32 clients and
synchronous commit off to maximize the amount of data inserted) with
the default vacuum settings. On master, after the first aggressive
vacuum, we do only anti-wraparound vacuums (and only two of these are
triggered). With the patch, after the first aggressive vacuum, 10 more
vacuums are triggered -- none of which are anti-wraparound vacuums.

I attached a chart comparing the autovacuums triggered on master vs
with the patch.

Besides the performance benefit of spreading the freezing work over
more normal vacuums (thereby disrupting foreground workloads less),
anti-wraparound vacuums are not auto canceled by DDL -- making them
more of a nuisance to users.

[1]
/messages/by-id/CAAKRu_aj-P7YyBz_cPNwztz6ohP+vWis=iz3YcomkB3NpYA--w@mail.gmail.com

#9Melanie Plageman
melanieplageman@gmail.com
In reply to: wenhui qiu (#8)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

Attached v6 is rebased over 306dc520b9dfd60

On Wed, Feb 5, 2025 at 8:54 PM wenhui qiu <qiuwenhuifx@gmail.com> wrote:

Hi Melanie Plageman
Thank you for working on this ,Actually, there were two patches aimed at optimizing vacuum-triggered processes, and one of them reached a consensus and has been committed:https://commitfest.postgresql.org/52/5046/ , https://commitfest.postgresql.org/51/5395/, Maybe referring to the already committed patch and setting a maximum value for vacuum_max_ins_threshold would be more acceptable.

We could add autovacuum_vacuum_insert_max_threshold, but with an
insert-only workload, we can expect that the cold data is being
frozen. By calculating the threshold based on unfrozen data, we are
effectively capping the threshold for inserted data without adding
another guc. If any of that data is being unfrozen via updates or
deletes, then the autovacuum_vacuum_max_threshold would apply.

Perhaps I'm missing a case where calculating the insert threshold on
unfrozen data would not act as a cap, in which case I could get on
board with a guc.

- Melanie

Attachments:

v6-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchapplication/x-patch; name=v6-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From 6048afbedf1590c3669c9f952fa70b244839c987 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v6 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++-----
 src/backend/postmaster/autovacuum.c           | 32 +++++++++++++++++--
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 39 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 38244409e3c..8cef54bf183 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8652,14 +8652,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is 0.2 (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 09ec9bb6990..250de2bb8ae 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2932,7 +2932,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3046,7 +3045,12 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+		int32		relallvisible = classForm->relallvisible;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3055,11 +3059,33 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If the table has been vacuumed and we have reliable data for
+		 * relallfrozen and relallvisible, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 *
+		 * If relallfrozen is -1, that means relallvisible was updated
+		 * manually and we can't rely on relallfrozen.
+		 */
+		if (relpages > 0 && reltuples > 0 && relallfrozen > -1)
+		{
+			if (relallvisible > relpages)
+				relallvisible = relpages;
+
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c40b7a3121e..543e365adb6 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -667,8 +667,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

v6-0001-Add-relallfrozen-to-pg_class.patchapplication/x-patch; name=v6-0001-Add-relallfrozen-to-pg_class.patchDownload
From f32c398378285d76ded53e90f19231dc20f960bb Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:19:57 -0500
Subject: [PATCH v6 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/catalogs.sgml              | 20 ++++++++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 +++++++++++++----
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 22 +++++++++++++++++-----
 src/backend/commands/analyze.c          | 12 ++++++------
 src/backend/commands/cluster.c          |  5 +++++
 src/backend/commands/vacuum.c           | 15 ++++++++++++++-
 src/backend/statistics/relation_stats.c | 23 ++++++++++++++++++++---
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/pg_class.h          |  8 ++++++++
 src/include/commands/vacuum.h           |  1 +
 11 files changed, 108 insertions(+), 19 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 088fb175cce..ba0ba0f55a3 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2064,6 +2064,26 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>,
+       and a few DDL commands such as
+       <link linkend="sql-createindex"><command>CREATE INDEX</command></link>.
+       <structfield>relallfrozen</structfield> must be less than or equal to
+       <structfield>relallvisible</structfield> as an all-frozen page must be
+       all-visible. If <structfield>relallvisible</structfield> was updated
+       manually, <structfield>relallfrozen</structfield> will be -1 until the
+       next time they are updated.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 075af385cd1..d7048fdfb71 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -370,7 +370,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -631,10 +632,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -643,7 +651,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3251,7 +3260,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 57ef466acce..5ab75d5c977 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -913,6 +913,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -983,6 +984,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 7377912b41e..ee72a1e5e41 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2792,8 +2792,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2811,6 +2811,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2850,7 +2851,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2875,8 +2882,8 @@ index_update_stats(Relation rel,
 	 * transaction could still fail before committing.  Setting relhasindex
 	 * true is safe even if there are no indexes (VACUUM will eventually fix
 	 * it).  And of course the new relpages and reltuples counts are correct
-	 * regardless.  However, we don't want to change relpages (or
-	 * relallvisible) if the caller isn't providing an updated reltuples
+	 * regardless. However, we don't want to change relpages (or relallvisible
+	 * and relallfrozen) if the caller isn't providing an updated reltuples
 	 * count, because that would bollix the reltuples/relpages ratio which is
 	 * what's really important.
 	 */
@@ -2923,6 +2930,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index f8da32e9aef..b9f17bf4212 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index e6745e6145c..376ce2e489a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1410,6 +1410,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1441,8 +1442,15 @@ vac_update_relstats(Relation relation,
 			 relid);
 	pgcform = (Form_pg_class) GETSTRUCT(ctup);
 
-	/* Apply statistical updates, if any, to copied tuple */
+	/*
+	 * An all-frozen block must be marked all-visible in the VM. While callers
+	 * are likely to check this themselves, it is worth ensuring we don't put
+	 * incorrect information in pg_class.
+	 */
+	if (num_all_frozen_pages > num_all_visible_pages)
+		num_all_frozen_pages = num_all_visible_pages;
 
+	/* Apply statistical updates, if any, to copied tuple */
 	dirty = false;
 	if (pgcform->relpages != (int32) num_pages)
 	{
@@ -1459,6 +1467,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 046661d7c3f..758a56f8a46 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -56,6 +56,10 @@ static bool relation_statistics_update(FunctionCallInfo fcinfo, int elevel,
 
 /*
  * Internal function for modifying statistics for a relation.
+ *
+ * Up to four pg_class columns may be updated even though only three relation
+ * statistics may be modified; relallfrozen is always set to -1 when
+ * relallvisible is updated manually.
  */
 static bool
 relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
@@ -167,6 +171,14 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		if (update_relallvisible && pgcform->relallvisible != relallvisible)
 		{
 			pgcform->relallvisible = relallvisible;
+
+			/*
+			 * If we are modifying relallvisible manually, it is not clear
+			 * what relallfrozen value would make sense. Therefore, set it to
+			 * -1 ("unknown"). It will be updated the next time these fields
+			 * are updated.
+			 */
+			pgcform->relallfrozen = -1;
 			dirty = true;
 		}
 
@@ -182,9 +194,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		TupleDesc	tupdesc = RelationGetDescr(crel);
 		HeapTuple	ctup;
 		Form_pg_class pgcform;
-		int			replaces[3] = {0};
-		Datum		values[3] = {0};
-		bool		nulls[3] = {0};
+		int			replaces[4] = {0};
+		Datum		values[4] = {0};
+		bool		nulls[4] = {0};
 		int			nreplaces = 0;
 
 		ctup = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
@@ -217,6 +229,11 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			replaces[nreplaces] = Anum_pg_class_relallvisible;
 			values[nreplaces] = Int32GetDatum(relallvisible);
 			nreplaces++;
+
+			/* See comment above in-place update of relallfrozen */
+			replaces[nreplaces] = Anum_pg_class_relallfrozen;
+			values[nreplaces] = Int32GetDatum(-1);
+			nreplaces++;
 		}
 
 		if (nreplaces > 0)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 43219a9629c..efd6bcb3860 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1925,6 +1925,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3882,6 +3883,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fbf42b0e195 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,14 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/*
+	 * # of all-frozen blocks (not always up-to-date)
+	 * Starts as 0 (if it has never been updated).
+	 * If relallvisible is manually updated, relallfrozen will be
+	 * -1, meaning "unknown"
+	 */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 12d0b61950d..b2a678df09e 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -329,6 +329,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.34.1

#10wenhui qiu
qiuwenhuifx@gmail.com
In reply to: Melanie Plageman (#9)
Re: Trigger more frequent autovacuums of heavy insert tables

Hi

We could add autovacuum_vacuum_insert_max_threshold, but with an
insert-only workload, we can expect that the cold data is being
frozen. By calculating the threshold based on unfrozen data, we are
effectively capping the threshold for inserted data without adding
another guc. If any of that data is being unfrozen via updates or
deletes, then the autovacuum_vacuum_max_threshold would apply.

Perhaps I'm missing a case where calculating the insert threshold on
unfrozen data would not act as a cap, in which case I could get on
board with a guc.

Actually ,I like your solution. Even I think this formula could use that
pcnt_unfrozen parameter
vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples *
pcnt_unfrozen;

Thanks

On Thu, Feb 6, 2025 at 11:42 PM Melanie Plageman <melanieplageman@gmail.com>
wrote:

Show quoted text

Attached v6 is rebased over 306dc520b9dfd60

On Wed, Feb 5, 2025 at 8:54 PM wenhui qiu <qiuwenhuifx@gmail.com> wrote:

Hi Melanie Plageman
Thank you for working on this ,Actually, there were two patches aimed

at optimizing vacuum-triggered processes, and one of them reached a
consensus and has been committed:
https://commitfest.postgresql.org/52/5046/ ,
https://commitfest.postgresql.org/51/5395/, Maybe referring to the
already committed patch and setting a maximum value for
vacuum_max_ins_threshold would be more acceptable.

We could add autovacuum_vacuum_insert_max_threshold, but with an
insert-only workload, we can expect that the cold data is being
frozen. By calculating the threshold based on unfrozen data, we are
effectively capping the threshold for inserted data without adding
another guc. If any of that data is being unfrozen via updates or
deletes, then the autovacuum_vacuum_max_threshold would apply.

Perhaps I'm missing a case where calculating the insert threshold on
unfrozen data would not act as a cap, in which case I could get on
board with a guc.

- Melanie

#11Japin Li
japinli@hotmail.com
In reply to: Melanie Plageman (#9)
Re: Trigger more frequent autovacuums of heavy insert tables

On Thu, 06 Feb 2025 at 10:42, Melanie Plageman <melanieplageman@gmail.com> wrote:

Attached v6 is rebased over 306dc520b9dfd60

On Wed, Feb 5, 2025 at 8:54 PM wenhui qiu <qiuwenhuifx@gmail.com> wrote:

Hi Melanie Plageman
Thank you for working on this ,Actually, there were two patches
aimed at optimizing vacuum-triggered processes, and one of them
reached a consensus and has been
committed:https://commitfest.postgresql.org/52/5046/ ,
https://commitfest.postgresql.org/51/5395/, Maybe referring to the
already committed patch and setting a maximum value for
vacuum_max_ins_threshold would be more acceptable.

We could add autovacuum_vacuum_insert_max_threshold, but with an
insert-only workload, we can expect that the cold data is being
frozen. By calculating the threshold based on unfrozen data, we are
effectively capping the threshold for inserted data without adding
another guc. If any of that data is being unfrozen via updates or
deletes, then the autovacuum_vacuum_max_threshold would apply.

Perhaps I'm missing a case where calculating the insert threshold on
unfrozen data would not act as a cap, in which case I could get on
board with a guc.

Make sense.

It appears that there is an incorrect indentation in the config.sgml file.
The <literal> is accidentally removed.

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 38244409e3c..571c73668f9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8652,10 +8652,10 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
+         Specifies a fraction of the active (unfrozen) table size to add to
          <varname>autovacuum_vacuum_insert_threshold</varname>
          when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
+         The default is <literal>0.2</literal> (20% of active table size).
          This parameter can only be set in the <filename>postgresql.conf</filename>
          file or on the server command line;
          but the setting can be overridden for individual tables by

--
Regrads,
Japin Li

#12Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#1)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Oct 22, 2024 at 03:12:53PM -0400, Melanie Plageman wrote:

By considering only the unfrozen portion of the table when calculating
the vacuum insert threshold, we can trigger vacuums more proactively
on insert-heavy tables. This changes the definition of
insert_scale_factor to a percentage of "active" table size. The
attached patch does this.

I think this is a creative idea. My first reaction is to question whether
it makes send to have two strategies for this sort of thing:
autovacuum_vacuum_max_threshold for updates/deletes and this for inserts.
Perhaps we don't want to more aggressively clean up bloat (except for the
very largest tables via the hard cap), but we do want to more aggressively
mark newly-inserted tuples frozen. I'm curious what you think.

I've estimated the unfrozen percentage of the table by adding a new
field to pg_class, relallfrozen, which is updated in the same places
as relallvisible.

Wouldn't relallvisible be sufficient here? We'll skip all-visible pages
unless this is an anti-wraparound vacuum, at which point I would think the
insert threshold goes out the window.

More frequent vacuums means each vacuum scans fewer pages, but, more
interestingly, the first vacuum after a checkpoint is much more
efficient. With the patch, the first vacuum after a checkpoint emits
half as many FPIs. You can see that only 18 pages were newly dirtied.
So, with the patch, the pages being vacuumed are usually still in
shared buffers and still dirty.

Are you aware of any scenarios where your proposed strategy might make
things worse? From your test results, it sounds like these vacuums ought
to usually be relatively efficient, so sending insert-only tables to the
front of the line is normally okay, but maybe that's not always true.

--
nathan

#13Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#12)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 7, 2025 at 12:37 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Tue, Oct 22, 2024 at 03:12:53PM -0400, Melanie Plageman wrote:

By considering only the unfrozen portion of the table when calculating
the vacuum insert threshold, we can trigger vacuums more proactively
on insert-heavy tables. This changes the definition of
insert_scale_factor to a percentage of "active" table size. The
attached patch does this.

I think this is a creative idea.

Indeed. I can't take much credit for it -- Andres suggested this
direction during an off-list conversation where I was complaining
about how difficult it was to benchmark my vacuum eager scanning patch
set [1] because normal vacuums were so rarely triggered for
insert-only tables after the first aggressive vacuum.

My first reaction is to question whether
it makes send to have two strategies for this sort of thing:
autovacuum_vacuum_max_threshold for updates/deletes and this for inserts.
Perhaps we don't want to more aggressively clean up bloat (except for the
very largest tables via the hard cap), but we do want to more aggressively
mark newly-inserted tuples frozen. I'm curious what you think.

The goal with insert-only tables is to set the whole page frozen in
the VM. So, the number of pages is more important than the total
number of tuples inserted. Whereas, with updates/deletes, it seems
like the total amount of garbage (# tuples) needing cleaning is more
important.

My intuition (maybe wrong) is that it is more common to have a bunch
of pages with a single (or few) updates/deletes than it is to have a
bunch of pages with a single insert. This patch is mostly meant to
trigger vacuums sooner on large insert-only or bulk loaded tables.
Though, it is more common to have a cluster of hot pages than
uniformly distributed updates and deletes...

I've estimated the unfrozen percentage of the table by adding a new
field to pg_class, relallfrozen, which is updated in the same places
as relallvisible.

Wouldn't relallvisible be sufficient here? We'll skip all-visible pages
unless this is an anti-wraparound vacuum, at which point I would think the
insert threshold goes out the window.

It's a great question. There are a couple reasons why I don't think so.

I think this might lead to triggering vacuums too often for
insert-mostly tables. For those tables, the pages that are not
all-visible will largely be just those with data that is new since the
last vacuum. And if we trigger vacuums based off of the % not
all-visible, we might decrease the number of cases where we are able
to vacuum inserted data and freeze it the first time it is vacuumed --
thereby increasing the total amount of work.

As for your point about us skipping all-visible pages except in
anti-wraparound vacuums -- that's not totally true. Autovacuums
triggered by the insert or update/delete thresholds and not by
autovacuum_freeze_max_age can also be aggressive (that's based on
vacuum_freeze_table_age). Aggressive vacuums scan all-visible pages.
And we actually want to trigger more normal aggressive (non-anti-wrap)
vacuums because anti-wraparound vacuums are not canceled by
conflicting lock requests (like those needed by DDL) -- see
PROC_VACUUM_FOR_WRAPAROUND in ProcSleep().

We also scan a surprising number of all-visible pages in practice due
to SKIP_PAGES_THRESHOLD. I was pretty taken aback while testing [1]
how many all-visible pages we scan due to this optimization. And, I'm
planning on merging [1] in the next few days, so this will also
increase the number of all-visible pages scanned during normal
vacuums.

More frequent vacuums means each vacuum scans fewer pages, but, more
interestingly, the first vacuum after a checkpoint is much more
efficient. With the patch, the first vacuum after a checkpoint emits
half as many FPIs. You can see that only 18 pages were newly dirtied.
So, with the patch, the pages being vacuumed are usually still in
shared buffers and still dirty.

Are you aware of any scenarios where your proposed strategy might make
things worse? From your test results, it sounds like these vacuums ought
to usually be relatively efficient, so sending insert-only tables to the
front of the line is normally okay, but maybe that's not always true.

So, of course they aren't exactly at the front of the line since we
autovacuum based on the order in pg_class. But, I suppose if you spend
a bunch of time vacuuming an insert-mostly table you previously would
have skipped instead of some other table -- that is effectively
prioritizing the insert-mostly tables.

For insert-only/mostly tables, what you are ideally doing is vacuuming
more frequently and handling a small number of pages each vacuum of
the relation, so it has a low performance impact. I suppose if you
only have a few autovacuum workers and an equal number of massive
insert-only tables, you could end up starving other actively updated
tables of vacuum resources. But, those insert-only tables would have
to be vacuumed eventually -- and I imagine that the impact of a
massive aggressive vacuum of all of the data in those tables would be
more disruptive than some extra bloat in your other tables.

I'd be interested if other people with more field experience can
imagine starvation scenarios that would be much worse with this patch.
What kinds of starvation scenarios do you normally see?

In terms of specific, dramatic differences in behavior (since this
wouldn't be hidden behind a guc) people might be surprised by how soon
tables start being vacuumed after a huge COPY FREEZE.

- Melanie

#14Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#13)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 07, 2025 at 02:21:07PM -0500, Melanie Plageman wrote:

On Fri, Feb 7, 2025 at 12:37 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

My first reaction is to question whether
it makes send to have two strategies for this sort of thing:
autovacuum_vacuum_max_threshold for updates/deletes and this for inserts.
Perhaps we don't want to more aggressively clean up bloat (except for the
very largest tables via the hard cap), but we do want to more aggressively
mark newly-inserted tuples frozen. I'm curious what you think.

The goal with insert-only tables is to set the whole page frozen in
the VM. So, the number of pages is more important than the total
number of tuples inserted. Whereas, with updates/deletes, it seems
like the total amount of garbage (# tuples) needing cleaning is more
important.

I think this is a reasonable position. To be clear, I don't have a problem
with having different strategies, or even with swapping
autovacuum_vacuum_max_threshold with a similar change, if it's the right
thing to do. I just want to be able to articulate why they're different.

Wouldn't relallvisible be sufficient here? We'll skip all-visible pages
unless this is an anti-wraparound vacuum, at which point I would think the
insert threshold goes out the window.

It's a great question. There are a couple reasons why I don't think so.

I think this might lead to triggering vacuums too often for
insert-mostly tables. For those tables, the pages that are not
all-visible will largely be just those with data that is new since the
last vacuum. And if we trigger vacuums based off of the % not
all-visible, we might decrease the number of cases where we are able
to vacuum inserted data and freeze it the first time it is vacuumed --
thereby increasing the total amount of work.

Rephrasing to make sure I understand correctly: you're saying that using
all-frozen would trigger less frequent insert vacuums, which would give us
a better chance of freezing more than more frequent insert vacuums
triggered via all-visible? My suspicion is that the difference would tend
to be quite subtle in practice, but I have no concrete evidence to back
that up.

--
nathan

#15Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#14)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 7, 2025 at 3:38 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Fri, Feb 07, 2025 at 02:21:07PM -0500, Melanie Plageman wrote:

On Fri, Feb 7, 2025 at 12:37 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

Wouldn't relallvisible be sufficient here? We'll skip all-visible pages
unless this is an anti-wraparound vacuum, at which point I would think the
insert threshold goes out the window.

It's a great question. There are a couple reasons why I don't think so.

I think this might lead to triggering vacuums too often for
insert-mostly tables. For those tables, the pages that are not
all-visible will largely be just those with data that is new since the
last vacuum. And if we trigger vacuums based off of the % not
all-visible, we might decrease the number of cases where we are able
to vacuum inserted data and freeze it the first time it is vacuumed --
thereby increasing the total amount of work.

Rephrasing to make sure I understand correctly: you're saying that using
all-frozen would trigger less frequent insert vacuums, which would give us
a better chance of freezing more than more frequent insert vacuums
triggered via all-visible? My suspicion is that the difference would tend
to be quite subtle in practice, but I have no concrete evidence to back
that up.

You understood me correctly.

As for relallfrozen, one of the justifications for adding it to
pg_class is actually for the visibility it would provide. We have no
way of knowing how many all-visible but not all-frozen pages there are
on users' systems without pg_visibility. If users had this
information, they could potentially tune their freeze-related settings
more aggressively. Regularly reading the whole visibility map with
pg_visibilitymap_summary() is pretty hard to justify on most
production systems. But querying pg_class every 10 minutes or
something is much more reasonable.

- Melanie

#16Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#15)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 07, 2025 at 03:57:49PM -0500, Melanie Plageman wrote:

As for relallfrozen, one of the justifications for adding it to
pg_class is actually for the visibility it would provide. We have no
way of knowing how many all-visible but not all-frozen pages there are
on users' systems without pg_visibility. If users had this
information, they could potentially tune their freeze-related settings
more aggressively. Regularly reading the whole visibility map with
pg_visibilitymap_summary() is pretty hard to justify on most
production systems. But querying pg_class every 10 minutes or
something is much more reasonable.

If we need it anyway, then I have no objections to using a freeze-related
metric for a freeze-related feature.

Okay, I'll actually look at the patches next...

--
nathan

#17Nathan Bossart
nathandbossart@gmail.com
In reply to: Nathan Bossart (#16)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 07, 2025 at 03:05:09PM -0600, Nathan Bossart wrote:

Okay, I'll actually look at the patches next...

Ugh, it's already been 10 days since I said that. A couple of thoughts on
0001:

I'm not sure I understand the reason for capping relallfrozen to
relallvisible. From upthread, I gather this is mostly to deal with manual
statistics manipulation, but my first reaction is that we should just let
those values be bogus. Is there something that fundamentally requires
relallfrozen to be <= relallvisible? These are only estimates, so I don't
think it would be that surprising for them to defy this expectation.

Should we allow manipulating relallfrozen like we do relallvisible? My
assumption is that would even be required for the ongoing statistics
import/export work.

Upthread, you mentioned that you weren't seeing relallfrozen in
pg_class_d.h. I checked on my machine and see it there as expected. Are
you still missing it?

--
nathan

#18Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#17)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Feb 17, 2025 at 11:11 AM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Fri, Feb 07, 2025 at 03:05:09PM -0600, Nathan Bossart wrote:

Okay, I'll actually look at the patches next...

Thanks for taking a look!

I'm not sure I understand the reason for capping relallfrozen to
relallvisible. From upthread, I gather this is mostly to deal with manual
statistics manipulation, but my first reaction is that we should just let
those values be bogus. Is there something that fundamentally requires
relallfrozen to be <= relallvisible? These are only estimates, so I don't
think it would be that surprising for them to defy this expectation.

I wasn't quite sure what to do here. I see your perspective: for
example, reltuples can't possibly be more than relpages but we don't
do any validation of that. My rationale wasn't exactly principled, so
I'll change it to not cap relallfrozen.

This makes me think I should also not cap relallfrozen when using it
in relation_needs_vacanalyze(). There I cap it to relallvisible and
relallvisible is capped to relpages. One of the ideas behind letting
people modify these stats in pg_class is that they can change a single
field to see what the effect on their system is, right?

Should we allow manipulating relallfrozen like we do relallvisible? My
assumption is that would even be required for the ongoing statistics
import/export work.

Why would it be required for the statistics import/export work?

Upthread, you mentioned that you weren't seeing relallfrozen in
pg_class_d.h. I checked on my machine and see it there as expected. Are
you still missing it?

I see it now. No idea what was happening.

- Melanie

#19Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#18)
Re: Trigger more frequent autovacuums of heavy insert tables

On Wed, Feb 19, 2025 at 04:36:05PM -0500, Melanie Plageman wrote:

This makes me think I should also not cap relallfrozen when using it
in relation_needs_vacanalyze(). There I cap it to relallvisible and
relallvisible is capped to relpages. One of the ideas behind letting
people modify these stats in pg_class is that they can change a single
field to see what the effect on their system is, right?

Right. Capping these values to reflect reality seems like it could make
that more difficult.

Should we allow manipulating relallfrozen like we do relallvisible? My
assumption is that would even be required for the ongoing statistics
import/export work.

Why would it be required for the statistics import/export work?

It's probably not strictly required, but my naive expectation would be that
we'd handle relallfrozen just like relallvisible, which appears to be
dumped in the latest stats import/export patch. Is there any reason we
shouldn't do the same for relallfrozen?

--
nathan

#20Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#19)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Wed, Feb 19, 2025 at 4:59 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Wed, Feb 19, 2025 at 04:36:05PM -0500, Melanie Plageman wrote:

This makes me think I should also not cap relallfrozen when using it
in relation_needs_vacanalyze(). There I cap it to relallvisible and
relallvisible is capped to relpages. One of the ideas behind letting
people modify these stats in pg_class is that they can change a single
field to see what the effect on their system is, right?

Right. Capping these values to reflect reality seems like it could make
that more difficult.

Attache v7 doesn't cap the result for manual stats updating done with
relation_statistics_update(). I did, however, keep the cap for the
places where vacuum/analyze/create index update the stats. There the
number for relallfrozen is coming directly from visibilitymap_count(),
so it should be correct. I could perhaps add an assert instead, but I
didn't think that really made sense. An assert is meant to help the
developer and what could the developer do about the visibility map
being corrupted.

Should we allow manipulating relallfrozen like we do relallvisible? My
assumption is that would even be required for the ongoing statistics
import/export work.

Why would it be required for the statistics import/export work?

It's probably not strictly required, but my naive expectation would be that
we'd handle relallfrozen just like relallvisible, which appears to be
dumped in the latest stats import/export patch. Is there any reason we
shouldn't do the same for relallfrozen?

Nope I don't think so, but I also don't know about how people are
envisioning using a manually updated relallvisible.

- Melanie

Attachments:

v7-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchtext/x-patch; charset=US-ASCII; name=v7-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From 26dac7d2d9768c072913c911776d4f679bb21126 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v7 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 007746a4429..cdd477e62c3 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8767,14 +8767,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ade2708b59e..050dbd43a22 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5362ff80519..acf45efc4ba 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -675,8 +675,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

v7-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v7-0001-Add-relallfrozen-to-pg_class.patchDownload
From 1437cb0669e3f2aea13ca34fab7618ecb58a8f14 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 20 Feb 2025 19:22:55 -0500
Subject: [PATCH v7 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Discussion: https://postgr.es/m/flat/Z7ZUWje-e1e_fKeu%40nathan#18ec7b0a585a16a58fe317b4ec42efe0
---
 doc/src/sgml/catalogs.sgml              | 22 ++++++++++++++
 src/backend/access/heap/vacuumlazy.c    | 17 ++++++++---
 src/backend/catalog/heap.c              |  2 ++
 src/backend/catalog/index.c             | 18 ++++++++++--
 src/backend/commands/analyze.c          | 19 +++++++++----
 src/backend/commands/cluster.c          |  5 ++++
 src/backend/commands/vacuum.c           |  6 ++++
 src/backend/statistics/relation_stats.c | 38 +++++++++++++++++++++++--
 src/backend/utils/cache/relcache.c      |  2 ++
 src/include/catalog/pg_class.h          |  3 ++
 src/include/commands/vacuum.h           |  1 +
 11 files changed, 117 insertions(+), 16 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index ee59a7e15d0..e371cd905f2 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,28 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few
+       DDL commands such as <link linkend="sql-createindex"><command>CREATE
+       INDEX</command></link>. Every all-frozen page must also be marked
+       all-visible in the visibility map, so internal processes will always
+       update <structfield>relallfrozen</structfield> to a value less than or
+       equal to <structfield>relallvisible</structfield>. However, if either
+       field is updated manually, it is possible for
+       <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>. This will be corrected the
+       next time these estimates are updated internally.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1af18a78a2b..ad01d7a2150 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -623,7 +623,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -898,10 +899,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -910,7 +918,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3720,7 +3729,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 956f196fc95..7ef6f0f1cba 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -924,6 +924,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -994,6 +995,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index cdabf780244..ac4d43fa3d3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2794,8 +2794,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2813,6 +2813,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2852,7 +2853,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2925,6 +2932,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd75954951b..9c4755a96d5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,18 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
+
+		/*
+		 * An all-frozen block _must_ be all-visible. As such, clamp the count
+		 * of all-frozen blocks to the count of all-visible blocks.
+		 */
+		if (relallfrozen > relallvisible)
+			relallfrozen = relallvisible;
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +652,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +669,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +685,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 0239d9bae65..e81c9a8aba3 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1427,6 +1427,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1476,6 +1477,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 046661d7c3f..e80fd52a876 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -27,6 +27,7 @@
 #define DEFAULT_RELPAGES Int32GetDatum(0)
 #define DEFAULT_RELTUPLES Float4GetDatum(-1.0)
 #define DEFAULT_RELALLVISIBLE Int32GetDatum(0)
+#define DEFAULT_RELALLFROZEN Int32GetDatum(0)
 
 /*
  * Positional argument numbers, names, and types for
@@ -39,6 +40,7 @@ enum relation_stats_argnum
 	RELPAGES_ARG,
 	RELTUPLES_ARG,
 	RELALLVISIBLE_ARG,
+	RELALLFROZEN_ARG,
 	NUM_RELATION_STATS_ARGS
 };
 
@@ -48,6 +50,7 @@ static struct StatsArgInfo relarginfo[] =
 	[RELPAGES_ARG] = {"relpages", INT4OID},
 	[RELTUPLES_ARG] = {"reltuples", FLOAT4OID},
 	[RELALLVISIBLE_ARG] = {"relallvisible", INT4OID},
+	[RELALLFROZEN_ARG] = {"relallfrozen", INT4OID},
 	[NUM_RELATION_STATS_ARGS] = {0}
 };
 
@@ -67,7 +70,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 	float		reltuples = DEFAULT_RELTUPLES;
 	bool		update_reltuples = false;
 	int32		relallvisible = DEFAULT_RELALLVISIBLE;
+	int32		relallfrozen = DEFAULT_RELALLFROZEN;
 	bool		update_relallvisible = false;
+	bool		update_relallfrozen = false;
 	bool		result = true;
 
 	if (!PG_ARGISNULL(RELPAGES_ARG))
@@ -120,6 +125,21 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			update_relallvisible = true;
 	}
 
+	if (!PG_ARGISNULL(RELALLFROZEN_ARG))
+	{
+		relallfrozen = PG_GETARG_INT32(RELALLFROZEN_ARG);
+
+		if (relallfrozen < 0)
+		{
+			ereport(elevel,
+					(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+					 errmsg("relallfrozen cannot be < 0")));
+			result = false;
+		}
+		else
+			update_relallfrozen = true;
+	}
+
 	stats_check_required_arg(fcinfo, relarginfo, RELATION_ARG);
 	reloid = PG_GETARG_OID(RELATION_ARG);
 
@@ -169,6 +189,11 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			pgcform->relallvisible = relallvisible;
 			dirty = true;
 		}
+		if (update_relallfrozen && pgcform->relallfrozen != relallfrozen)
+		{
+			pgcform->relallfrozen = relallfrozen;
+			dirty = true;
+		}
 
 		if (dirty)
 			systable_inplace_update_finish(inplace_state, ctup);
@@ -182,9 +207,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		TupleDesc	tupdesc = RelationGetDescr(crel);
 		HeapTuple	ctup;
 		Form_pg_class pgcform;
-		int			replaces[3] = {0};
-		Datum		values[3] = {0};
-		bool		nulls[3] = {0};
+		int			replaces[4] = {0};
+		Datum		values[4] = {0};
+		bool		nulls[4] = {0};
 		int			nreplaces = 0;
 
 		ctup = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
@@ -219,6 +244,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			nreplaces++;
 		}
 
+		if (update_relallfrozen && relallfrozen != pgcform->relallfrozen)
+		{
+			replaces[nreplaces] = Anum_pg_class_relallfrozen;
+			values[nreplaces] = Int32GetDatum(relallfrozen);
+			nreplaces++;
+		}
+
 		if (nreplaces > 0)
 		{
 			HeapTuple	newtup;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 398114373e9..d1ae761b3f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1928,6 +1928,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3885,6 +3886,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fa96ba07bf4 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1571a66c6bf..baacc63f590 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -349,6 +349,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
-- 
2.34.1

#21Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#20)
Re: Trigger more frequent autovacuums of heavy insert tables

On Thu, Feb 20, 2025 at 07:35:32PM -0500, Melanie Plageman wrote:

Attache v7 doesn't cap the result for manual stats updating done with
relation_statistics_update().

Unfortunately, this already needs a rebase...

I did, however, keep the cap for the
places where vacuum/analyze/create index update the stats. There the
number for relallfrozen is coming directly from visibilitymap_count(),
so it should be correct. I could perhaps add an assert instead, but I
didn't think that really made sense. An assert is meant to help the
developer and what could the developer do about the visibility map
being corrupted.

This might just be personal preference, but I think this is exactly the
sort of thing an assertion is meant for. If we expect the value to always
be correct, and it's not, then our assumptions were wrong or someone has
broken something. In both of these cases, I as a developer would really
like to know so that I don't introduce a latent bug. If we want Postgres
to gracefully handle or detect visibility map corruption, then maybe we
should do both or PANIC.

I do see that heap_vacuum_rel() already caps relallvisible to relpages, but
it's not clear to me whether that's something that we expect to regularly
happen in practice or what the adverse effects might be. So perhaps I'm
misunderstanding the scope and severity of bogus results from
visibilitymap_count(). Commit e6858e6, which added this code, doesn't say
anything about safety, but commit 3d351d9 changed the comment in question
to its current wording. After a very quick skim of the latter's thread
[0]: /messages/by-id/flat/F02298E0-6EF4-49A1-BCB6-C484794D9ACC@thebuild.com

Should we allow manipulating relallfrozen like we do relallvisible? My
assumption is that would even be required for the ongoing statistics
import/export work.

Why would it be required for the statistics import/export work?

It's probably not strictly required, but my naive expectation would be that
we'd handle relallfrozen just like relallvisible, which appears to be
dumped in the latest stats import/export patch. Is there any reason we
shouldn't do the same for relallfrozen?

Nope I don't think so, but I also don't know about how people are
envisioning using a manually updated relallvisible.

That does seem unlikely. I'd expect it to be more useful for porting
statistics over during pg_upgrade.

[0]: /messages/by-id/flat/F02298E0-6EF4-49A1-BCB6-C484794D9ACC@thebuild.com

--
nathan

#22Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#21)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Feb 24, 2025 at 4:53 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Thu, Feb 20, 2025 at 07:35:32PM -0500, Melanie Plageman wrote:

Attache v7 doesn't cap the result for manual stats updating done with
relation_statistics_update().

Unfortunately, this already needs a rebase...

Thanks! Attached v8 does just that.

I've also taken a pass at updating the stats import tests to include
relallfrozen. I'm not totally sure how I feel about the results. I
erred on the side of putting relallfrozen wherever relallvisible was.
But, that means relallfrozen is included in test cases where it
doesn't seem important to the test case. But, that was true with
relallvisible too.

Anyway, I talked to Corey off-list last week. Maybe he will have a
chance to weigh in on the test cases. Also, I had forgotten to include
relallfrozen in pg_clear_relation_stats(). I've fixed that in v8, but
now I'm worried there are some other places I may have missed.

I did, however, keep the cap for the
places where vacuum/analyze/create index update the stats. There the
number for relallfrozen is coming directly from visibilitymap_count(),
so it should be correct. I could perhaps add an assert instead, but I
didn't think that really made sense. An assert is meant to help the
developer and what could the developer do about the visibility map
being corrupted.

This might just be personal preference, but I think this is exactly the
sort of thing an assertion is meant for. If we expect the value to always
be correct, and it's not, then our assumptions were wrong or someone has
broken something. In both of these cases, I as a developer would really
like to know so that I don't introduce a latent bug. If we want Postgres
to gracefully handle or detect visibility map corruption, then maybe we
should do both or PANIC.

I'm on the fence about adding a PANIC. We do PANIC in other places
where we notice corruption (like PageAddItemExtended()). But, in most
of the cases, it seems like we are PANICing because there isn't a
reasonable way to accomplish the intended task. In this case, we
probably can't trust the visibility map counts for that page, but the
pg_class columns are just estimates, so just capping relallfrozen
might be good enough.

I will note that in the other place where we may notice corruption in
the VM, in lazy_scan_prune(), we do visibilitymap_clear() and print a
WARNING -- as opposed to PANICing. Perhaps that's because there is no
need to panic, since we are already fixing the problem with
visibiliytmap_clear().

An assert would only help if the developer did something while
developing that corrupted the visibility map. It doesn't help keep
bogus values out of pg_class if a user's visibility map got corrupted
in some way. But, maybe that isn't needed.

I do see that heap_vacuum_rel() already caps relallvisible to relpages, but
it's not clear to me whether that's something that we expect to regularly
happen in practice or what the adverse effects might be. So perhaps I'm
misunderstanding the scope and severity of bogus results from
visibilitymap_count(). Commit e6858e6, which added this code, doesn't say
anything about safety, but commit 3d351d9 changed the comment in question
to its current wording. After a very quick skim of the latter's thread
[0], I don't see any discussion about this point, either.

Thanks for doing this archaeology.

I am hesitant to keep the current cap on relallvisible in
heap_vacuum_rel() but then not include an equivalent cap for
relallfrozen. And I think it would be even more confusing for
relallvisible to be capped but relallfrozen has an assert instead.

None of the other locations where relallvisible is updated
(do_analyze_rel(), index_update_stats()) do this capping of
relallvisible. It seems like we should make it consistent. Perhaps we
should just remove it from heap_vacuum_rel(). Then add an assert in
all these places to at least protect development mistakes.

- Melanie

Attachments:

v8-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v8-0001-Add-relallfrozen-to-pg_class.patchDownload
From eda45d2beb8bc79a117d252467f343047b33a06d Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 24 Feb 2025 17:27:18 -0500
Subject: [PATCH v8 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Discussion: https://postgr.es/m/flat/Z7ZUWje-e1e_fKeu%40nathan#18ec7b0a585a16a58fe317b4ec42efe0
---
 doc/src/sgml/catalogs.sgml                 |  22 +++
 src/backend/access/heap/vacuumlazy.c       |  17 +-
 src/backend/catalog/heap.c                 |   2 +
 src/backend/catalog/index.c                |  18 ++-
 src/backend/commands/analyze.c             |  19 ++-
 src/backend/commands/cluster.c             |   5 +
 src/backend/commands/vacuum.c              |   6 +
 src/backend/statistics/relation_stats.c    |  34 +++-
 src/backend/utils/cache/relcache.c         |   2 +
 src/include/catalog/pg_class.h             |   3 +
 src/include/catalog/pg_proc.dat            |   4 +-
 src/include/commands/vacuum.h              |   1 +
 src/test/regress/expected/stats_import.out | 175 +++++++++++++--------
 src/test/regress/sql/stats_import.sql      |  85 +++++++---
 14 files changed, 282 insertions(+), 111 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index ee59a7e15d0..e371cd905f2 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,28 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few
+       DDL commands such as <link linkend="sql-createindex"><command>CREATE
+       INDEX</command></link>. Every all-frozen page must also be marked
+       all-visible in the visibility map, so internal processes will always
+       update <structfield>relallfrozen</structfield> to a value less than or
+       equal to <structfield>relallvisible</structfield>. However, if either
+       field is updated manually, it is possible for
+       <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>. This will be corrected the
+       next time these estimates are updated internally.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1af18a78a2b..ad01d7a2150 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -623,7 +623,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -898,10 +899,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -910,7 +918,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3720,7 +3729,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 956f196fc95..7ef6f0f1cba 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -924,6 +924,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -994,6 +995,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index f37b990c81d..e878541f432 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2793,8 +2793,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2812,6 +2812,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2851,7 +2852,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2924,6 +2931,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd75954951b..9c4755a96d5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,18 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
+
+		/*
+		 * An all-frozen block _must_ be all-visible. As such, clamp the count
+		 * of all-frozen blocks to the count of all-visible blocks.
+		 */
+		if (relallfrozen > relallvisible)
+			relallfrozen = relallvisible;
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +652,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +669,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +685,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 0239d9bae65..e81c9a8aba3 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1427,6 +1427,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1476,6 +1477,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 1f6c5c39121..b3aaf73791c 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -36,6 +36,7 @@ enum relation_stats_argnum
 	RELPAGES_ARG,
 	RELTUPLES_ARG,
 	RELALLVISIBLE_ARG,
+	RELALLFROZEN_ARG,
 	NUM_RELATION_STATS_ARGS
 };
 
@@ -45,6 +46,7 @@ static struct StatsArgInfo relarginfo[] =
 	[RELPAGES_ARG] = {"relpages", INT4OID},
 	[RELTUPLES_ARG] = {"reltuples", FLOAT4OID},
 	[RELALLVISIBLE_ARG] = {"relallvisible", INT4OID},
+	[RELALLFROZEN_ARG] = {"relallfrozen", INT4OID},
 	[NUM_RELATION_STATS_ARGS] = {0}
 };
 
@@ -66,6 +68,8 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 	bool		update_reltuples = false;
 	BlockNumber relallvisible = 0;
 	bool		update_relallvisible = false;
+	BlockNumber relallfrozen = 0;
+	bool		update_relallfrozen = false;
 
 	if (!PG_ARGISNULL(RELPAGES_ARG))
 	{
@@ -93,6 +97,12 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		update_relallvisible = true;
 	}
 
+	if (!PG_ARGISNULL(RELALLFROZEN_ARG))
+	{
+		relallfrozen = PG_GETARG_UINT32(RELALLFROZEN_ARG);
+		update_relallfrozen = true;
+	}
+
 	stats_check_required_arg(fcinfo, relarginfo, RELATION_ARG);
 	reloid = PG_GETARG_OID(RELATION_ARG);
 
@@ -142,6 +152,11 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			pgcform->relallvisible = relallvisible;
 			dirty = true;
 		}
+		if (update_relallfrozen && pgcform->relallfrozen != relallfrozen)
+		{
+			pgcform->relallfrozen = relallfrozen;
+			dirty = true;
+		}
 
 		if (dirty)
 			systable_inplace_update_finish(inplace_state, ctup);
@@ -155,9 +170,9 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 		TupleDesc	tupdesc = RelationGetDescr(crel);
 		HeapTuple	ctup;
 		Form_pg_class pgcform;
-		int			replaces[3] = {0};
-		Datum		values[3] = {0};
-		bool		nulls[3] = {0};
+		int			replaces[4] = {0};
+		Datum		values[4] = {0};
+		bool		nulls[4] = {0};
 		int			nreplaces = 0;
 
 		ctup = SearchSysCache1(RELOID, ObjectIdGetDatum(reloid));
@@ -192,6 +207,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel, bool inplace)
 			nreplaces++;
 		}
 
+		if (update_relallfrozen && relallfrozen != pgcform->relallfrozen)
+		{
+			replaces[nreplaces] = Anum_pg_class_relallfrozen;
+			values[nreplaces] = UInt32GetDatum(relallfrozen);
+			nreplaces++;
+		}
+
 		if (nreplaces > 0)
 		{
 			HeapTuple	newtup;
@@ -230,9 +252,9 @@ pg_set_relation_stats(PG_FUNCTION_ARGS)
 Datum
 pg_clear_relation_stats(PG_FUNCTION_ARGS)
 {
-	LOCAL_FCINFO(newfcinfo, 4);
+	LOCAL_FCINFO(newfcinfo, 5);
 
-	InitFunctionCallInfoData(*newfcinfo, NULL, 4, InvalidOid, NULL, NULL);
+	InitFunctionCallInfoData(*newfcinfo, NULL, 5, InvalidOid, NULL, NULL);
 
 	newfcinfo->args[0].value = PG_GETARG_OID(0);
 	newfcinfo->args[0].isnull = PG_ARGISNULL(0);
@@ -242,6 +264,8 @@ pg_clear_relation_stats(PG_FUNCTION_ARGS)
 	newfcinfo->args[2].isnull = false;
 	newfcinfo->args[3].value = UInt32GetDatum(0);
 	newfcinfo->args[3].isnull = false;
+	newfcinfo->args[4].value = UInt32GetDatum(0);
+	newfcinfo->args[4].isnull = false;
 
 	relation_statistics_update(newfcinfo, ERROR, false);
 	PG_RETURN_VOID();
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 398114373e9..d1ae761b3f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1928,6 +1928,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3885,6 +3886,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fa96ba07bf4 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index af9546de23d..a1b51cc973a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12447,8 +12447,8 @@
   descr => 'set statistics on relation',
   proname => 'pg_set_relation_stats', provolatile => 'v', proisstrict => 'f',
   proparallel => 'u', prorettype => 'void',
-  proargtypes => 'regclass int4 float4 int4',
-  proargnames => '{relation,relpages,reltuples,relallvisible}',
+  proargtypes => 'regclass int4 float4 int4 int4',
+  proargnames => '{relation,relpages,reltuples,relallvisible,relallfrozen}',
   prosrc => 'pg_set_relation_stats' },
 { oid => '9945',
   descr => 'clear statistics on relation',
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1571a66c6bf..baacc63f590 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -349,6 +349,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
diff --git a/src/test/regress/expected/stats_import.out b/src/test/regress/expected/stats_import.out
index 0e8491131e3..a88622f6680 100644
--- a/src/test/regress/expected/stats_import.out
+++ b/src/test/regress/expected/stats_import.out
@@ -13,12 +13,12 @@ CREATE TABLE stats_import.test(
     tags text[]
 ) WITH (autovacuum_enabled = false);
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- error: regclass not found
@@ -27,7 +27,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 ERROR:  could not open relation with OID 0
 -- relpages default
 SELECT
@@ -35,7 +36,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -47,7 +49,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -59,7 +62,21 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+ pg_set_relation_stats 
+-----------------------
+ 
+(1 row)
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -71,18 +88,19 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 CREATE INDEX test_i ON stats_import.test(id);
@@ -129,18 +147,19 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       18 |       401 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       18 |       401 |             5 |            3
 (1 row)
 
 -- test MVCC behavior: changes do not persist after abort (in contrast
@@ -151,19 +170,20 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 4000.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
 ABORT;
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       18 |       401 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       18 |       401 |             5 |            3
 (1 row)
 
 BEGIN;
@@ -176,12 +196,12 @@ SELECT
 (1 row)
 
 ABORT;
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       18 |       401 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       18 |       401 |             5 |            3
 (1 row)
 
 -- clear
@@ -193,12 +213,12 @@ SELECT
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- invalid relkinds for statistics
@@ -788,7 +808,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 is NULL
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -796,7 +817,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 has type "integer", expected type "text"
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -804,6 +826,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 ERROR:  variadic arguments must be name/value pairs
 HINT:  Provide an even number of variadic arguments that can be divided into pairs.
@@ -813,7 +836,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  could not open relation with OID 0
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -821,18 +845,19 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
  pg_restore_relation_stats 
 ---------------------------
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 -- ok: just relpages
@@ -845,12 +870,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       15 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       15 |       400 |             4 |            2
 (1 row)
 
 -- test non-MVCC behavior: new value should persist after abort
@@ -865,12 +890,12 @@ SELECT pg_restore_relation_stats(
 (1 row)
 
 ABORT;
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- ok: just reltuples
@@ -883,12 +908,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             4 |            2
 (1 row)
 
 -- ok: just relallvisible
@@ -901,12 +926,30 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            2
+(1 row)
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+ pg_restore_relation_stats 
+---------------------------
+ t
+(1 row)
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            3
 (1 row)
 
 -- warn and error: unrecognized argument name
@@ -924,19 +967,20 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 WARNING:  argument "relpages" has type "text", expected type "integer"
  pg_restore_relation_stats 
 ---------------------------
  f
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- error: object does not exist
@@ -1510,7 +1554,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
  pg_restore_relation_stats 
 ---------------------------
@@ -1694,12 +1739,12 @@ WHERE s.starelid = 'stats_import.is_odd'::regclass;
 (0 rows)
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        1 |         4 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        1 |         4 |             0 |            0
 (1 row)
 
 --
diff --git a/src/test/regress/sql/stats_import.sql b/src/test/regress/sql/stats_import.sql
index 5e24c779d80..a95ec7608ba 100644
--- a/src/test/regress/sql/stats_import.sql
+++ b/src/test/regress/sql/stats_import.sql
@@ -16,7 +16,7 @@ CREATE TABLE stats_import.test(
 ) WITH (autovacuum_enabled = false);
 
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -26,7 +26,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relpages default
 SELECT
@@ -34,7 +35,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- reltuples default
 SELECT
@@ -42,7 +44,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relallvisible default
 SELECT
@@ -50,7 +53,17 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
 
 -- named arguments
 SELECT
@@ -58,9 +71,10 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -94,9 +108,10 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -108,10 +123,11 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 4000.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 ABORT;
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -121,7 +137,7 @@ SELECT
         'stats_import.test'::regclass);
 ABORT;
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -130,7 +146,7 @@ SELECT
     pg_catalog.pg_clear_relation_stats(
         'stats_import.test'::regclass);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -610,7 +626,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -618,7 +635,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -626,6 +644,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 
 -- reject: object doesn't exist
@@ -634,7 +653,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -642,9 +662,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -654,7 +675,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '15'::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -666,7 +687,7 @@ SELECT pg_restore_relation_stats(
         'relpages', '16'::integer);
 ABORT;
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -676,7 +697,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'reltuples', '500'::real);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -686,7 +707,17 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relallvisible', 5::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -704,9 +735,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -1127,7 +1159,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
 
 -- Generate statistics on table with data
@@ -1282,7 +1315,7 @@ JOIN pg_attribute a ON a.attrelid = s.starelid AND a.attnum = s.staattnum
 WHERE s.starelid = 'stats_import.is_odd'::regclass;
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
-- 
2.34.1

v8-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchtext/x-patch; charset=US-ASCII; name=v8-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From c950c451a0e9d89854a94fcf86a5159982456cd9 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v8 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a8354576108..9d8e42cd3db 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8745,14 +8745,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ddb303f5201..0aca7d78b90 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e771d87da1f..13bd844b66c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -674,8 +674,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

#23wenhui qiu
qiuwenhuifx@gmail.com
In reply to: Melanie Plageman (#22)
Re: Trigger more frequent autovacuums of heavy insert tables

Hi Melanie

relallvisible. It seems like we should make it consistent. Perhaps we
should just remove it from heap_vacuum_rel(). Then add an assert in
all these places to at least protect development mistakes.

I think there's some objection to that.

Thanks

On Tue, Feb 25, 2025 at 7:35 AM Melanie Plageman <melanieplageman@gmail.com>
wrote:

Show quoted text

On Mon, Feb 24, 2025 at 4:53 PM Nathan Bossart <nathandbossart@gmail.com>
wrote:

On Thu, Feb 20, 2025 at 07:35:32PM -0500, Melanie Plageman wrote:

Attache v7 doesn't cap the result for manual stats updating done with
relation_statistics_update().

Unfortunately, this already needs a rebase...

Thanks! Attached v8 does just that.

I've also taken a pass at updating the stats import tests to include
relallfrozen. I'm not totally sure how I feel about the results. I
erred on the side of putting relallfrozen wherever relallvisible was.
But, that means relallfrozen is included in test cases where it
doesn't seem important to the test case. But, that was true with
relallvisible too.

Anyway, I talked to Corey off-list last week. Maybe he will have a
chance to weigh in on the test cases. Also, I had forgotten to include
relallfrozen in pg_clear_relation_stats(). I've fixed that in v8, but
now I'm worried there are some other places I may have missed.

I did, however, keep the cap for the
places where vacuum/analyze/create index update the stats. There the
number for relallfrozen is coming directly from visibilitymap_count(),
so it should be correct. I could perhaps add an assert instead, but I
didn't think that really made sense. An assert is meant to help the
developer and what could the developer do about the visibility map
being corrupted.

This might just be personal preference, but I think this is exactly the
sort of thing an assertion is meant for. If we expect the value to

always

be correct, and it's not, then our assumptions were wrong or someone has
broken something. In both of these cases, I as a developer would really
like to know so that I don't introduce a latent bug. If we want Postgres
to gracefully handle or detect visibility map corruption, then maybe we
should do both or PANIC.

I'm on the fence about adding a PANIC. We do PANIC in other places
where we notice corruption (like PageAddItemExtended()). But, in most
of the cases, it seems like we are PANICing because there isn't a
reasonable way to accomplish the intended task. In this case, we
probably can't trust the visibility map counts for that page, but the
pg_class columns are just estimates, so just capping relallfrozen
might be good enough.

I will note that in the other place where we may notice corruption in
the VM, in lazy_scan_prune(), we do visibilitymap_clear() and print a
WARNING -- as opposed to PANICing. Perhaps that's because there is no
need to panic, since we are already fixing the problem with
visibiliytmap_clear().

An assert would only help if the developer did something while
developing that corrupted the visibility map. It doesn't help keep
bogus values out of pg_class if a user's visibility map got corrupted
in some way. But, maybe that isn't needed.

I do see that heap_vacuum_rel() already caps relallvisible to relpages,

but

it's not clear to me whether that's something that we expect to regularly
happen in practice or what the adverse effects might be. So perhaps I'm
misunderstanding the scope and severity of bogus results from
visibilitymap_count(). Commit e6858e6, which added this code, doesn't

say

anything about safety, but commit 3d351d9 changed the comment in question
to its current wording. After a very quick skim of the latter's thread
[0], I don't see any discussion about this point, either.

Thanks for doing this archaeology.

I am hesitant to keep the current cap on relallvisible in
heap_vacuum_rel() but then not include an equivalent cap for
relallfrozen. And I think it would be even more confusing for
relallvisible to be capped but relallfrozen has an assert instead.

None of the other locations where relallvisible is updated
(do_analyze_rel(), index_update_stats()) do this capping of
relallvisible. It seems like we should make it consistent. Perhaps we
should just remove it from heap_vacuum_rel(). Then add an assert in
all these places to at least protect development mistakes.

- Melanie

#24Melanie Plageman
melanieplageman@gmail.com
In reply to: wenhui qiu (#23)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Feb 24, 2025 at 10:30 PM wenhui qiu <qiuwenhuifx@gmail.com> wrote:

Hi Melanie

relallvisible. It seems like we should make it consistent. Perhaps we
should just remove it from heap_vacuum_rel(). Then add an assert in
all these places to at least protect development mistakes.

I think there's some objection to that.

Could you elaborate a bit?

There were new merge conflicts, so v9 is attached.

- Melanie

Attachments:

v9-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v9-0001-Add-relallfrozen-to-pg_class.patchDownload
From de1d6c8779a3c4082f41d74f25854970ed1a6190 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 24 Feb 2025 17:27:18 -0500
Subject: [PATCH v9 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Discussion: https://postgr.es/m/flat/Z7ZUWje-e1e_fKeu%40nathan#18ec7b0a585a16a58fe317b4ec42efe0
---
 doc/src/sgml/catalogs.sgml                 |  22 +++
 src/backend/access/heap/vacuumlazy.c       |  17 ++-
 src/backend/catalog/heap.c                 |   2 +
 src/backend/catalog/index.c                |  18 ++-
 src/backend/catalog/system_functions.sql   |   3 +-
 src/backend/commands/analyze.c             |  19 ++-
 src/backend/commands/cluster.c             |   5 +
 src/backend/commands/vacuum.c              |   6 +
 src/backend/statistics/relation_stats.c    |  29 +++-
 src/backend/utils/cache/relcache.c         |   2 +
 src/include/catalog/pg_class.h             |   3 +
 src/include/catalog/pg_proc.dat            |   4 +-
 src/include/commands/vacuum.h              |   1 +
 src/test/regress/expected/stats_import.out | 148 +++++++++++++--------
 src/test/regress/sql/stats_import.sql      |  76 ++++++++---
 15 files changed, 260 insertions(+), 95 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index ee59a7e15d0..e371cd905f2 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,28 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few
+       DDL commands such as <link linkend="sql-createindex"><command>CREATE
+       INDEX</command></link>. Every all-frozen page must also be marked
+       all-visible in the visibility map, so internal processes will always
+       update <structfield>relallfrozen</structfield> to a value less than or
+       equal to <structfield>relallvisible</structfield>. However, if either
+       field is updated manually, it is possible for
+       <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>. This will be corrected the
+       next time these estimates are updated internally.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1af18a78a2b..ad01d7a2150 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -623,7 +623,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -898,10 +899,17 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -910,7 +918,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3720,7 +3729,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 956f196fc95..7ef6f0f1cba 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -924,6 +924,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -994,6 +995,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index f37b990c81d..e878541f432 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2793,8 +2793,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2812,6 +2812,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2851,7 +2852,13 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+		{
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
+
+			/* An all-frozen block must be all-visible in the VM */
+			if (relallfrozen > relallvisible)
+				relallfrozen = relallvisible;
+		}
 	}
 
 	/*
@@ -2924,6 +2931,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 591157b1d1b..d0f84323e0a 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -640,7 +640,8 @@ CREATE OR REPLACE FUNCTION
   pg_set_relation_stats(relation regclass,
                         relpages integer DEFAULT NULL,
                         reltuples real DEFAULT NULL,
-                        relallvisible integer DEFAULT NULL)
+                        relallvisible integer DEFAULT NULL,
+                        relallfrozen integer DEFAULT NULL)
 RETURNS void
 LANGUAGE INTERNAL
 CALLED ON NULL INPUT VOLATILE
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd75954951b..9c4755a96d5 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,18 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
+
+		/*
+		 * An all-frozen block _must_ be all-visible. As such, clamp the count
+		 * of all-frozen blocks to the count of all-visible blocks.
+		 */
+		if (relallfrozen > relallvisible)
+			relallfrozen = relallvisible;
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +652,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +669,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +685,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 0239d9bae65..e81c9a8aba3 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1427,6 +1427,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1476,6 +1477,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 66731290a3e..2d5fb4efddb 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -36,6 +36,7 @@ enum relation_stats_argnum
 	RELPAGES_ARG,
 	RELTUPLES_ARG,
 	RELALLVISIBLE_ARG,
+	RELALLFROZEN_ARG,
 	NUM_RELATION_STATS_ARGS
 };
 
@@ -45,6 +46,7 @@ static struct StatsArgInfo relarginfo[] =
 	[RELPAGES_ARG] = {"relpages", INT4OID},
 	[RELTUPLES_ARG] = {"reltuples", FLOAT4OID},
 	[RELALLVISIBLE_ARG] = {"relallvisible", INT4OID},
+	[RELALLFROZEN_ARG] = {"relallfrozen", INT4OID},
 	[NUM_RELATION_STATS_ARGS] = {0}
 };
 
@@ -65,11 +67,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 	bool		update_reltuples = false;
 	BlockNumber relallvisible = 0;
 	bool		update_relallvisible = false;
+	BlockNumber relallfrozen = 0;
+	bool		update_relallfrozen = false;
 	HeapTuple	ctup;
 	Form_pg_class pgcform;
-	int			replaces[3] = {0};
-	Datum		values[3] = {0};
-	bool		nulls[3] = {0};
+	int			replaces[4] = {0};
+	Datum		values[4] = {0};
+	bool		nulls[4] = {0};
 	int			nreplaces = 0;
 
 	if (!PG_ARGISNULL(RELPAGES_ARG))
@@ -98,6 +102,12 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 		update_relallvisible = true;
 	}
 
+	if (!PG_ARGISNULL(RELALLFROZEN_ARG))
+	{
+		relallfrozen = PG_GETARG_UINT32(RELALLFROZEN_ARG);
+		update_relallfrozen = true;
+	}
+
 	stats_check_required_arg(fcinfo, relarginfo, RELATION_ARG);
 	reloid = PG_GETARG_OID(RELATION_ARG);
 
@@ -148,6 +158,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 		nreplaces++;
 	}
 
+	if (update_relallfrozen && relallfrozen != pgcform->relallfrozen)
+	{
+		replaces[nreplaces] = Anum_pg_class_relallfrozen;
+		values[nreplaces] = UInt32GetDatum(relallfrozen);
+		nreplaces++;
+	}
+
 	if (nreplaces > 0)
 	{
 		TupleDesc	tupdesc = RelationGetDescr(crel);
@@ -186,9 +203,9 @@ pg_set_relation_stats(PG_FUNCTION_ARGS)
 Datum
 pg_clear_relation_stats(PG_FUNCTION_ARGS)
 {
-	LOCAL_FCINFO(newfcinfo, 4);
+	LOCAL_FCINFO(newfcinfo, 5);
 
-	InitFunctionCallInfoData(*newfcinfo, NULL, 4, InvalidOid, NULL, NULL);
+	InitFunctionCallInfoData(*newfcinfo, NULL, 5, InvalidOid, NULL, NULL);
 
 	newfcinfo->args[0].value = PG_GETARG_OID(0);
 	newfcinfo->args[0].isnull = PG_ARGISNULL(0);
@@ -198,6 +215,8 @@ pg_clear_relation_stats(PG_FUNCTION_ARGS)
 	newfcinfo->args[2].isnull = false;
 	newfcinfo->args[3].value = UInt32GetDatum(0);
 	newfcinfo->args[3].isnull = false;
+	newfcinfo->args[4].value = UInt32GetDatum(0);
+	newfcinfo->args[4].isnull = false;
 
 	relation_statistics_update(newfcinfo, ERROR);
 	PG_RETURN_VOID();
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 398114373e9..d1ae761b3f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1928,6 +1928,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3885,6 +3886,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fa96ba07bf4 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index af9546de23d..a1b51cc973a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12447,8 +12447,8 @@
   descr => 'set statistics on relation',
   proname => 'pg_set_relation_stats', provolatile => 'v', proisstrict => 'f',
   proparallel => 'u', prorettype => 'void',
-  proargtypes => 'regclass int4 float4 int4',
-  proargnames => '{relation,relpages,reltuples,relallvisible}',
+  proargtypes => 'regclass int4 float4 int4 int4',
+  proargnames => '{relation,relpages,reltuples,relallvisible,relallfrozen}',
   prosrc => 'pg_set_relation_stats' },
 { oid => '9945',
   descr => 'clear statistics on relation',
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1571a66c6bf..baacc63f590 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -349,6 +349,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
diff --git a/src/test/regress/expected/stats_import.out b/src/test/regress/expected/stats_import.out
index d6713eacc2c..944fec03050 100644
--- a/src/test/regress/expected/stats_import.out
+++ b/src/test/regress/expected/stats_import.out
@@ -13,12 +13,12 @@ CREATE TABLE stats_import.test(
     tags text[]
 ) WITH (autovacuum_enabled = false);
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- error: regclass not found
@@ -27,7 +27,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 ERROR:  could not open relation with OID 0
 -- relpages default
 SELECT
@@ -35,7 +36,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -47,7 +49,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -59,7 +62,21 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+ pg_set_relation_stats 
+-----------------------
+ 
+(1 row)
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -71,18 +88,19 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 CREATE INDEX test_i ON stats_import.test(id);
@@ -129,18 +147,19 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       18 |       401 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       18 |       401 |             5 |            3
 (1 row)
 
 SELECT relpages, reltuples, relallvisible
@@ -160,12 +179,12 @@ SELECT
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- invalid relkinds for statistics
@@ -755,7 +774,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 is NULL
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -763,7 +783,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 has type "integer", expected type "text"
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -771,6 +792,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 ERROR:  variadic arguments must be name/value pairs
 HINT:  Provide an even number of variadic arguments that can be divided into pairs.
@@ -780,7 +802,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  could not open relation with OID 0
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -788,18 +811,19 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
  pg_restore_relation_stats 
 ---------------------------
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 -- ok: just relpages
@@ -812,12 +836,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- ok: just reltuples
@@ -830,12 +854,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             4 |            2
 (1 row)
 
 -- ok: just relallvisible
@@ -848,12 +872,30 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            2
+(1 row)
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+ pg_restore_relation_stats 
+---------------------------
+ t
+(1 row)
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            3
 (1 row)
 
 -- warn and error: unrecognized argument name
@@ -871,19 +913,20 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 WARNING:  argument "relpages" has type "text", expected type "integer"
  pg_restore_relation_stats 
 ---------------------------
  f
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- error: object does not exist
@@ -1457,7 +1500,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
  pg_restore_relation_stats 
 ---------------------------
@@ -1641,12 +1685,12 @@ WHERE s.starelid = 'stats_import.is_odd'::regclass;
 (0 rows)
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        1 |         4 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        1 |         4 |             0 |            0
 (1 row)
 
 --
diff --git a/src/test/regress/sql/stats_import.sql b/src/test/regress/sql/stats_import.sql
index 9740ab3ff02..dc909003784 100644
--- a/src/test/regress/sql/stats_import.sql
+++ b/src/test/regress/sql/stats_import.sql
@@ -16,7 +16,7 @@ CREATE TABLE stats_import.test(
 ) WITH (autovacuum_enabled = false);
 
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -26,7 +26,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relpages default
 SELECT
@@ -34,7 +35,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- reltuples default
 SELECT
@@ -42,7 +44,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relallvisible default
 SELECT
@@ -50,7 +53,17 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
 
 -- named arguments
 SELECT
@@ -58,9 +71,10 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -94,9 +108,10 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -109,7 +124,7 @@ SELECT
     pg_catalog.pg_clear_relation_stats(
         'stats_import.test'::regclass);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -589,7 +604,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -597,7 +613,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -605,6 +622,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 
 -- reject: object doesn't exist
@@ -613,7 +631,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -621,9 +640,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -633,7 +653,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '16'::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -643,7 +663,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'reltuples', '500'::real);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -653,7 +673,17 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relallvisible', 5::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -671,9 +701,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -1094,7 +1125,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
 
 -- Generate statistics on table with data
@@ -1249,7 +1281,7 @@ JOIN pg_attribute a ON a.attrelid = s.starelid AND a.attnum = s.staattnum
 WHERE s.starelid = 'stats_import.is_odd'::regclass;
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
-- 
2.34.1

v9-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchtext/x-patch; charset=US-ASCII; name=v9-0002-Trigger-more-frequent-autovacuums-with-relallfroz.patchDownload
From 59f301f9145622a91908c63b88354785ecfb69b2 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v9 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a8354576108..9d8e42cd3db 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8745,14 +8745,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ddb303f5201..0aca7d78b90 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e771d87da1f..13bd844b66c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -674,8 +674,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

#25Robert Haas
robertmhaas@gmail.com
In reply to: Melanie Plageman (#22)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Feb 24, 2025 at 6:35 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I'm on the fence about adding a PANIC. We do PANIC in other places
where we notice corruption (like PageAddItemExtended()). But, in most
of the cases, it seems like we are PANICing because there isn't a
reasonable way to accomplish the intended task. In this case, we
probably can't trust the visibility map counts for that page, but the
pg_class columns are just estimates, so just capping relallfrozen
might be good enough.

+1 for not unnecessarily inflating the log level. I agree with the
principle you articulate here: a PANIC is reasonable when there's no
sane way to continue, but for other corruption events, a WARNING is
better. Note that one really bad consequence of using ERROR or any
higher level in VACUUM is that VACUUM doesn't ever complete. We just
keep retrying and dying. Even if it's only an ERROR, now we're no
longer controlling bloat or preventing wraparound. That's extremely
painful for users, so we don't want to end up there if we have a
reasonable alternative.

Also, I think that it's usually a bad idea to use an Assert to test
for data corruption scenarios. The problem is that programmers are
more or less entitled to assume that an assertion will always hold
true, barring bugs, and just ignore completely what the code might do
if the assertion turns out to be false. But data does get corrupted on
disk, so you SHOULD think about what the code is going to do in such
scenarios. Depending on the details, you might want to WARNING or
ERROR or PANIC or try to repair it or try to just tolerate it being
wrong or something else -- but you should be thinking about what the
code is actually going to do, not just going "eh, well, that can't
happen".

--
Robert Haas
EDB: http://www.enterprisedb.com

#26Melanie Plageman
melanieplageman@gmail.com
In reply to: Robert Haas (#25)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 10:39 AM Robert Haas <robertmhaas@gmail.com> wrote:

On Mon, Feb 24, 2025 at 6:35 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I'm on the fence about adding a PANIC. We do PANIC in other places
where we notice corruption (like PageAddItemExtended()). But, in most
of the cases, it seems like we are PANICing because there isn't a
reasonable way to accomplish the intended task. In this case, we
probably can't trust the visibility map counts for that page, but the
pg_class columns are just estimates, so just capping relallfrozen
might be good enough.

+1 for not unnecessarily inflating the log level. I agree with the
principle you articulate here: a PANIC is reasonable when there's no
sane way to continue, but for other corruption events, a WARNING is
better. Note that one really bad consequence of using ERROR or any
higher level in VACUUM is that VACUUM doesn't ever complete. We just
keep retrying and dying. Even if it's only an ERROR, now we're no
longer controlling bloat or preventing wraparound. That's extremely
painful for users, so we don't want to end up there if we have a
reasonable alternative.

That makes sense. I think I'll add a WARNING if we don't cap the value.

Also, I think that it's usually a bad idea to use an Assert to test
for data corruption scenarios. The problem is that programmers are
more or less entitled to assume that an assertion will always hold
true, barring bugs, and just ignore completely what the code might do
if the assertion turns out to be false. But data does get corrupted on
disk, so you SHOULD think about what the code is going to do in such
scenarios. Depending on the details, you might want to WARNING or
ERROR or PANIC or try to repair it or try to just tolerate it being
wrong or something else -- but you should be thinking about what the
code is actually going to do, not just going "eh, well, that can't
happen".

I agree with this. I don't see what the assertion would catch in this
case. Even as I try to think of a scenario where this would help the
developer, if you write code that corrupts the visibility map, I don't
think waiting for an assert at the end of a vacuum will be your first
or best signal.

This does however leave me with the question of how to handle the
original question of whether or not to cap the proposed relallfrozen
to the value of relallvisible when updating stats at the end of
vacuum. The current code in heap_vacuum_rel() caps relallvisible to
relpages, so capping relallfrozen to relallvisible would follow that
pattern. However, the other places relallvisible is updated do no such
capping (do_analyze_rel(), index_update_stats()). It doesn't seem like
there is a good reason to do it one place and not the others. So, I
suggest either removing all the caps and adding a WARNING or capping
the value in all places. Because users can now manually update these
values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus
value due to manual statistics intervention. This led me to think that
a WARNING and no cap would be more effective for heap_vacuum_rel().

- Melanie

#27Nathan Bossart
nathandbossart@gmail.com
In reply to: Robert Haas (#25)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 10:38:48AM -0500, Robert Haas wrote:

On Mon, Feb 24, 2025 at 6:35 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

I'm on the fence about adding a PANIC. We do PANIC in other places
where we notice corruption (like PageAddItemExtended()). But, in most
of the cases, it seems like we are PANICing because there isn't a
reasonable way to accomplish the intended task. In this case, we
probably can't trust the visibility map counts for that page, but the
pg_class columns are just estimates, so just capping relallfrozen
might be good enough.

+1 for not unnecessarily inflating the log level. I agree with the
principle you articulate here: a PANIC is reasonable when there's no
sane way to continue, but for other corruption events, a WARNING is
better. Note that one really bad consequence of using ERROR or any
higher level in VACUUM is that VACUUM doesn't ever complete. We just
keep retrying and dying. Even if it's only an ERROR, now we're no
longer controlling bloat or preventing wraparound. That's extremely
painful for users, so we don't want to end up there if we have a
reasonable alternative.

Also, I think that it's usually a bad idea to use an Assert to test
for data corruption scenarios. The problem is that programmers are
more or less entitled to assume that an assertion will always hold
true, barring bugs, and just ignore completely what the code might do
if the assertion turns out to be false. But data does get corrupted on
disk, so you SHOULD think about what the code is going to do in such
scenarios. Depending on the details, you might want to WARNING or
ERROR or PANIC or try to repair it or try to just tolerate it being
wrong or something else -- but you should be thinking about what the
code is actually going to do, not just going "eh, well, that can't
happen".

In this case, it sounds to me like we want production builds to tolerate
visibility map corruption (at least, to some extent), but we don't really
expect to see it regularly. If that's true, then my inclination would be
to add assertions to make sure nobody has fundamentally broken the code,
but to at most either warn or restrict the values for non-assert builds.
TBH my gut feeling is that, outside of assert-enabled builds, we should
just let the values be bogus if they are indeed just estimates that can be
wildly inaccurate anyway. I see little value in trying to gently corral
the value in certain cases, lest we want to mask bugs and give credence to
bad assumptions about the possible values.

I think the argument for PANIC-ing in this case is that visibility map
corruption seems likely to be the tip of the iceberg. If we see that one
file is corrupted, I would expect a DBA to be extremely wary of trusting
the other files on the system, too. (I will freely admit that I might be
missing situations where race conditions or the like cause us to retrieve
seemingly invalid values for relpages and relallvisible, so please pardon
my ignorance as I dive deeper into this code.)

--
nathan

#28Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#26)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 11:02:40AM -0500, Melanie Plageman wrote:

This does however leave me with the question of how to handle the
original question of whether or not to cap the proposed relallfrozen
to the value of relallvisible when updating stats at the end of
vacuum. The current code in heap_vacuum_rel() caps relallvisible to
relpages, so capping relallfrozen to relallvisible would follow that
pattern. However, the other places relallvisible is updated do no such
capping (do_analyze_rel(), index_update_stats()). It doesn't seem like
there is a good reason to do it one place and not the others. So, I
suggest either removing all the caps and adding a WARNING or capping
the value in all places. Because users can now manually update these
values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus
value due to manual statistics intervention. This led me to think that
a WARNING and no cap would be more effective for heap_vacuum_rel().

I'm currently leaning towards the "remove all caps" idea. But I'm not sure
I totally understand the need for a WARNING. What do we expect users to do
with that information? If it's intended to alert them of possible
corruption, then IMHO a WARNING may be too gentle. I guess we could warn
and suggest a way to fix the value with the new statistics manipulation
functions if it's not that big of a deal, but at that point we might as
well just cap it on our own again. If we don't really expect users to have
to do anything about it, then isn't it just adding unnecessary log noise?

--
nathan

#29Greg Sabino Mullane
htamfids@gmail.com
In reply to: Melanie Plageman (#26)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 11:03 AM Melanie Plageman <melanieplageman@gmail.com>
wrote:

Because users can now manually update these values in pg_class, there
wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus value
due to manual statistics intervention.

Er..you had me until this. If manual monkeying of the system catalogs leads
to a "bogus" error that resembles a real one, then sow the wind, and reap
the whirlwind. I don't think that should be a consideration here.

--
Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

#30Robert Treat
rob@xzilla.net
In reply to: Nathan Bossart (#28)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 11:32 AM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Tue, Feb 25, 2025 at 11:02:40AM -0500, Melanie Plageman wrote:

This does however leave me with the question of how to handle the
original question of whether or not to cap the proposed relallfrozen
to the value of relallvisible when updating stats at the end of
vacuum. The current code in heap_vacuum_rel() caps relallvisible to
relpages, so capping relallfrozen to relallvisible would follow that
pattern. However, the other places relallvisible is updated do no such
capping (do_analyze_rel(), index_update_stats()). It doesn't seem like
there is a good reason to do it one place and not the others. So, I
suggest either removing all the caps and adding a WARNING or capping
the value in all places. Because users can now manually update these
values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus
value due to manual statistics intervention. This led me to think that
a WARNING and no cap would be more effective for heap_vacuum_rel().

I'm currently leaning towards the "remove all caps" idea. But I'm not sure
I totally understand the need for a WARNING. What do we expect users to do
with that information? If it's intended to alert them of possible
corruption, then IMHO a WARNING may be too gentle. I guess we could warn
and suggest a way to fix the value with the new statistics manipulation
functions if it's not that big of a deal, but at that point we might as
well just cap it on our own again. If we don't really expect users to have
to do anything about it, then isn't it just adding unnecessary log noise?

If the end user is manipulating numbers to test some theory, I think
it's valuable feedback that they have probably bent the system too
far, because we are now seeing numbers that don't make sense. If they
aren't mucking with the system, then it's valuable feedback that they
may have an underlying system problem that could be about to get
worse.

Robert Treat
https://xzilla.net

#31Nathan Bossart
nathandbossart@gmail.com
In reply to: Robert Treat (#30)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 12:36:40PM -0500, Robert Treat wrote:

On Tue, Feb 25, 2025 at 11:32 AM Nathan Bossart
<nathandbossart@gmail.com> wrote:

I'm currently leaning towards the "remove all caps" idea. But I'm not sure
I totally understand the need for a WARNING. What do we expect users to do
with that information? If it's intended to alert them of possible
corruption, then IMHO a WARNING may be too gentle. I guess we could warn
and suggest a way to fix the value with the new statistics manipulation
functions if it's not that big of a deal, but at that point we might as
well just cap it on our own again. If we don't really expect users to have
to do anything about it, then isn't it just adding unnecessary log noise?

If the end user is manipulating numbers to test some theory, I think
it's valuable feedback that they have probably bent the system too
far, because we are now seeing numbers that don't make sense.

If we do that in other cases, then that seems reasonable. But it feels
weird to me to carve out just this one inaccuracy. For example, do we warn
if someone sets reltuples too high for relpages, or if they set relpages
too low for reltuples? I'm not seeing why the relallfrozen <=
relallvisible <= relpages case is especially important to uphold. If we
were consistent about enforcing these kinds of invariants everywhere, then
I think I would be more on board with capping the values and/or ERROR-ing
when folks tried to set bogus ones. But if the only outcome of bogus
values is that you might get weird plans or autovacuum might prioritize the
table differently, then I'm not sure I see the point. You can accomplish
both of those things with totally valid values already.

If they
aren't mucking with the system, then it's valuable feedback that they
may have an underlying system problem that could be about to get
worse.

Maybe a WARNING is as much as we can realistically do in this case, but I
think it could be easily missed in the server logs. I dunno, it just feels
wrong to me to deal with potential corruption by gently notifying the user
and proceeding normally. I guess that's what we do already elsewhere,
though (e.g., for relfrozenxid/relminmxid in vac_update_relstats()).

--
nathan

#32Robert Haas
robertmhaas@gmail.com
In reply to: Melanie Plageman (#26)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 11:03 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

This does however leave me with the question of how to handle the
original question of whether or not to cap the proposed relallfrozen
to the value of relallvisible when updating stats at the end of
vacuum. The current code in heap_vacuum_rel() caps relallvisible to
relpages, so capping relallfrozen to relallvisible would follow that
pattern. However, the other places relallvisible is updated do no such
capping (do_analyze_rel(), index_update_stats()). It doesn't seem like
there is a good reason to do it one place and not the others. So, I
suggest either removing all the caps and adding a WARNING or capping
the value in all places. Because users can now manually update these
values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus
value due to manual statistics intervention. This led me to think that
a WARNING and no cap would be more effective for heap_vacuum_rel().

I mean, does it really make any difference one way or the other?

Given that users could manually update the catalog, we have to be able
to tolerate bad data in the catalogs without the world ending. If that
code has to exist anyway, then it's not mandatory to cap. On the other
hand, there's no great virtue in refusing to correct data that we know
to be wrong. Unless there is some other consideration which makes one
way better than the other, this feels like author's choice.

--
Robert Haas
EDB: http://www.enterprisedb.com

#33Nathan Bossart
nathandbossart@gmail.com
In reply to: Robert Haas (#32)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 01:52:28PM -0500, Robert Haas wrote:

Given that users could manually update the catalog, we have to be able
to tolerate bad data in the catalogs without the world ending. If that
code has to exist anyway, then it's not mandatory to cap. On the other
hand, there's no great virtue in refusing to correct data that we know
to be wrong. Unless there is some other consideration which makes one
way better than the other, this feels like author's choice.

Maybe the most conservative choice is to simply follow the example of
surrounding code. If it's careful to cap relallvisible to relpages, also
have it cap relallfrozen. If not, don't. *shrug*

In any case, I don't want to hold up this patch on this relatively minor
point. This seems like something we could pretty easily change in the
future if needed.

--
nathan

#34Melanie Plageman
melanieplageman@gmail.com
In reply to: Greg Sabino Mullane (#29)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 11:36 AM Greg Sabino Mullane <htamfids@gmail.com> wrote:

On Tue, Feb 25, 2025 at 11:03 AM Melanie Plageman <melanieplageman@gmail.com> wrote:

Because users can now manually update these values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus value due to manual statistics intervention.

Er..you had me until this. If manual monkeying of the system catalogs leads to a "bogus" error that resembles a real one, then sow the wind, and reap the whirlwind. I don't think that should be a consideration here.

Oh, the WARNING would only show up in a case of actual VM corruption.
The WARNING proposed is after calling visibilitymap_count() before
updating pg_class, if the value we get from the VM for relallfrozen
exceeds relallvisible. If you manually change pg_class
relallfrozen/relallvisible to bogus values, you wouldn't get a
warning. I meant there wouldn't be a way to detect the difference when
viewing pg_class between bogus values because of a corrupt VM and
bogus values because of manual updates to pg_class.

- Melanie

#35Melanie Plageman
melanieplageman@gmail.com
In reply to: Robert Haas (#32)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 1:52 PM Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Feb 25, 2025 at 11:03 AM Melanie Plageman
<melanieplageman@gmail.com> wrote:

This does however leave me with the question of how to handle the
original question of whether or not to cap the proposed relallfrozen
to the value of relallvisible when updating stats at the end of
vacuum. The current code in heap_vacuum_rel() caps relallvisible to
relpages, so capping relallfrozen to relallvisible would follow that
pattern. However, the other places relallvisible is updated do no such
capping (do_analyze_rel(), index_update_stats()). It doesn't seem like
there is a good reason to do it one place and not the others. So, I
suggest either removing all the caps and adding a WARNING or capping
the value in all places. Because users can now manually update these
values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus
value due to manual statistics intervention. This led me to think that
a WARNING and no cap would be more effective for heap_vacuum_rel().

I mean, does it really make any difference one way or the other?

Given that users could manually update the catalog, we have to be able
to tolerate bad data in the catalogs without the world ending. If that
code has to exist anyway, then it's not mandatory to cap. On the other
hand, there's no great virtue in refusing to correct data that we know
to be wrong. Unless there is some other consideration which makes one
way better than the other, this feels like author's choice.

I realized that whether or not we add a WARNING is an independent
question from whether or not we cap these values. In these instances,
we happen to have just read the whole VM and so we can tell you if it
is broken in a particular way. If I want to write a patch to warn
users of visibility map corruption after calling
visibilitymap_count(), I could do that and it might be a good idea,
but it should probably be a separate commit anyway.

- Melanie

#36Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#33)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 3:05 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Tue, Feb 25, 2025 at 01:52:28PM -0500, Robert Haas wrote:

Given that users could manually update the catalog, we have to be able
to tolerate bad data in the catalogs without the world ending. If that
code has to exist anyway, then it's not mandatory to cap. On the other
hand, there's no great virtue in refusing to correct data that we know
to be wrong. Unless there is some other consideration which makes one
way better than the other, this feels like author's choice.

Maybe the most conservative choice is to simply follow the example of
surrounding code. If it's careful to cap relallvisible to relpages, also
have it cap relallfrozen. If not, don't. *shrug*

Agreed. I've done this in attached v10. I handle relallfrozen values >
relpages in the second patch in the set when using the relallfrozen
value, so I think we are all good.

In any case, I don't want to hold up this patch on this relatively minor
point. This seems like something we could pretty easily change in the
future if needed.

Yes, so one thing you haven't said yet is if you are +1 on going
forward with these patches in general.

As for the code, I'm not 100% convinced I've got all the stats
import/export bits perfect (those are changing under my feet right now
anyway).

- Melanie

Attachments:

v10-0002-Trigger-more-frequent-autovacuums-with-relallfro.patchtext/x-patch; charset=US-ASCII; name=v10-0002-Trigger-more-frequent-autovacuums-with-relallfro.patchDownload
From bdeafa6bffbcbb895ceba5acc8433695ef1e29bf Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v10 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a8354576108..9d8e42cd3db 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8745,14 +8745,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ddb303f5201..0aca7d78b90 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e771d87da1f..13bd844b66c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -674,8 +674,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

v10-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v10-0001-Add-relallfrozen-to-pg_class.patchDownload
From b73b959ffaef4a7da6f297042273b32b5ba23c25 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 24 Feb 2025 17:27:18 -0500
Subject: [PATCH v10 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Discussion: https://postgr.es/m/flat/Z7ZUWje-e1e_fKeu%40nathan#18ec7b0a585a16a58fe317b4ec42efe0
---
 doc/src/sgml/catalogs.sgml                 |  21 +++
 src/backend/access/heap/vacuumlazy.c       |  18 ++-
 src/backend/catalog/heap.c                 |   2 +
 src/backend/catalog/index.c                |  12 +-
 src/backend/catalog/system_functions.sql   |   3 +-
 src/backend/commands/analyze.c             |  12 +-
 src/backend/commands/cluster.c             |   5 +
 src/backend/commands/vacuum.c              |   6 +
 src/backend/statistics/relation_stats.c    |  29 +++-
 src/backend/utils/cache/relcache.c         |   2 +
 src/include/catalog/pg_class.h             |   3 +
 src/include/catalog/pg_proc.dat            |   4 +-
 src/include/commands/vacuum.h              |   1 +
 src/test/regress/expected/stats_import.out | 148 +++++++++++++--------
 src/test/regress/sql/stats_import.sql      |  76 ++++++++---
 15 files changed, 247 insertions(+), 95 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index ee59a7e15d0..bdf27724585 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,27 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few
+       DDL commands such as <link linkend="sql-createindex"><command>CREATE
+       INDEX</command></link>. Every all-frozen page must also be marked
+       all-visible in the visibility map, so
+       <structfield>relallfrozen</structfield> should be less than or equal to
+       <structfield>relallvisible</structfield>. However, if either field is
+       updated manually or if the visibility map is corrupted, it is possible
+       for <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1af18a78a2b..3b91d02605a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -623,7 +623,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -898,10 +899,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks. This matches the
+	 * clamping of relallvisible above.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -910,7 +919,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3720,7 +3730,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 956f196fc95..7ef6f0f1cba 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -924,6 +924,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -994,6 +995,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index f37b990c81d..8e1741c81f5 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2793,8 +2793,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2812,6 +2812,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2851,7 +2852,7 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
 	}
 
 	/*
@@ -2924,6 +2925,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 591157b1d1b..d0f84323e0a 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -640,7 +640,8 @@ CREATE OR REPLACE FUNCTION
   pg_set_relation_stats(relation regclass,
                         relpages integer DEFAULT NULL,
                         reltuples real DEFAULT NULL,
-                        relallvisible integer DEFAULT NULL)
+                        relallvisible integer DEFAULT NULL,
+                        relallfrozen integer DEFAULT NULL)
 RETURNS void
 LANGUAGE INTERNAL
 CALLED ON NULL INPUT VOLATILE
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd75954951b..2b5fbdcbd82 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 0239d9bae65..e81c9a8aba3 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1427,6 +1427,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1476,6 +1477,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 66731290a3e..2d5fb4efddb 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -36,6 +36,7 @@ enum relation_stats_argnum
 	RELPAGES_ARG,
 	RELTUPLES_ARG,
 	RELALLVISIBLE_ARG,
+	RELALLFROZEN_ARG,
 	NUM_RELATION_STATS_ARGS
 };
 
@@ -45,6 +46,7 @@ static struct StatsArgInfo relarginfo[] =
 	[RELPAGES_ARG] = {"relpages", INT4OID},
 	[RELTUPLES_ARG] = {"reltuples", FLOAT4OID},
 	[RELALLVISIBLE_ARG] = {"relallvisible", INT4OID},
+	[RELALLFROZEN_ARG] = {"relallfrozen", INT4OID},
 	[NUM_RELATION_STATS_ARGS] = {0}
 };
 
@@ -65,11 +67,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 	bool		update_reltuples = false;
 	BlockNumber relallvisible = 0;
 	bool		update_relallvisible = false;
+	BlockNumber relallfrozen = 0;
+	bool		update_relallfrozen = false;
 	HeapTuple	ctup;
 	Form_pg_class pgcform;
-	int			replaces[3] = {0};
-	Datum		values[3] = {0};
-	bool		nulls[3] = {0};
+	int			replaces[4] = {0};
+	Datum		values[4] = {0};
+	bool		nulls[4] = {0};
 	int			nreplaces = 0;
 
 	if (!PG_ARGISNULL(RELPAGES_ARG))
@@ -98,6 +102,12 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 		update_relallvisible = true;
 	}
 
+	if (!PG_ARGISNULL(RELALLFROZEN_ARG))
+	{
+		relallfrozen = PG_GETARG_UINT32(RELALLFROZEN_ARG);
+		update_relallfrozen = true;
+	}
+
 	stats_check_required_arg(fcinfo, relarginfo, RELATION_ARG);
 	reloid = PG_GETARG_OID(RELATION_ARG);
 
@@ -148,6 +158,13 @@ relation_statistics_update(FunctionCallInfo fcinfo, int elevel)
 		nreplaces++;
 	}
 
+	if (update_relallfrozen && relallfrozen != pgcform->relallfrozen)
+	{
+		replaces[nreplaces] = Anum_pg_class_relallfrozen;
+		values[nreplaces] = UInt32GetDatum(relallfrozen);
+		nreplaces++;
+	}
+
 	if (nreplaces > 0)
 	{
 		TupleDesc	tupdesc = RelationGetDescr(crel);
@@ -186,9 +203,9 @@ pg_set_relation_stats(PG_FUNCTION_ARGS)
 Datum
 pg_clear_relation_stats(PG_FUNCTION_ARGS)
 {
-	LOCAL_FCINFO(newfcinfo, 4);
+	LOCAL_FCINFO(newfcinfo, 5);
 
-	InitFunctionCallInfoData(*newfcinfo, NULL, 4, InvalidOid, NULL, NULL);
+	InitFunctionCallInfoData(*newfcinfo, NULL, 5, InvalidOid, NULL, NULL);
 
 	newfcinfo->args[0].value = PG_GETARG_OID(0);
 	newfcinfo->args[0].isnull = PG_ARGISNULL(0);
@@ -198,6 +215,8 @@ pg_clear_relation_stats(PG_FUNCTION_ARGS)
 	newfcinfo->args[2].isnull = false;
 	newfcinfo->args[3].value = UInt32GetDatum(0);
 	newfcinfo->args[3].isnull = false;
+	newfcinfo->args[4].value = UInt32GetDatum(0);
+	newfcinfo->args[4].isnull = false;
 
 	relation_statistics_update(newfcinfo, ERROR);
 	PG_RETURN_VOID();
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 398114373e9..d1ae761b3f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1928,6 +1928,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3885,6 +3886,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fa96ba07bf4 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index af9546de23d..a1b51cc973a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -12447,8 +12447,8 @@
   descr => 'set statistics on relation',
   proname => 'pg_set_relation_stats', provolatile => 'v', proisstrict => 'f',
   proparallel => 'u', prorettype => 'void',
-  proargtypes => 'regclass int4 float4 int4',
-  proargnames => '{relation,relpages,reltuples,relallvisible}',
+  proargtypes => 'regclass int4 float4 int4 int4',
+  proargnames => '{relation,relpages,reltuples,relallvisible,relallfrozen}',
   prosrc => 'pg_set_relation_stats' },
 { oid => '9945',
   descr => 'clear statistics on relation',
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1571a66c6bf..baacc63f590 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -349,6 +349,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
diff --git a/src/test/regress/expected/stats_import.out b/src/test/regress/expected/stats_import.out
index d6713eacc2c..944fec03050 100644
--- a/src/test/regress/expected/stats_import.out
+++ b/src/test/regress/expected/stats_import.out
@@ -13,12 +13,12 @@ CREATE TABLE stats_import.test(
     tags text[]
 ) WITH (autovacuum_enabled = false);
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- error: regclass not found
@@ -27,7 +27,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 ERROR:  could not open relation with OID 0
 -- relpages default
 SELECT
@@ -35,7 +36,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -47,7 +49,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -59,7 +62,21 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+ pg_set_relation_stats 
+-----------------------
+ 
+(1 row)
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
  pg_set_relation_stats 
 -----------------------
  
@@ -71,18 +88,19 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 CREATE INDEX test_i ON stats_import.test(id);
@@ -129,18 +147,19 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
  pg_set_relation_stats 
 -----------------------
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       18 |       401 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       18 |       401 |             5 |            3
 (1 row)
 
 SELECT relpages, reltuples, relallvisible
@@ -160,12 +179,12 @@ SELECT
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 -- invalid relkinds for statistics
@@ -755,7 +774,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 is NULL
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -763,7 +783,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  name at variadic position 5 has type "integer", expected type "text"
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -771,6 +792,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 ERROR:  variadic arguments must be name/value pairs
 HINT:  Provide an even number of variadic arguments that can be divided into pairs.
@@ -780,7 +802,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 ERROR:  could not open relation with OID 0
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -788,18 +811,19 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
  pg_restore_relation_stats 
 ---------------------------
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 -- ok: just relpages
@@ -812,12 +836,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- ok: just reltuples
@@ -830,12 +854,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             4 |            2
 (1 row)
 
 -- ok: just relallvisible
@@ -848,12 +872,30 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            2
+(1 row)
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+ pg_restore_relation_stats 
+---------------------------
+ t
+(1 row)
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            3
 (1 row)
 
 -- warn and error: unrecognized argument name
@@ -871,19 +913,20 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 WARNING:  argument "relpages" has type "text", expected type "integer"
  pg_restore_relation_stats 
 ---------------------------
  f
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- error: object does not exist
@@ -1457,7 +1500,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
  pg_restore_relation_stats 
 ---------------------------
@@ -1641,12 +1685,12 @@ WHERE s.starelid = 'stats_import.is_odd'::regclass;
 (0 rows)
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        1 |         4 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        1 |         4 |             0 |            0
 (1 row)
 
 --
diff --git a/src/test/regress/sql/stats_import.sql b/src/test/regress/sql/stats_import.sql
index 9740ab3ff02..dc909003784 100644
--- a/src/test/regress/sql/stats_import.sql
+++ b/src/test/regress/sql/stats_import.sql
@@ -16,7 +16,7 @@ CREATE TABLE stats_import.test(
 ) WITH (autovacuum_enabled = false);
 
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -26,7 +26,8 @@ SELECT
         relation => 0::Oid,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relpages default
 SELECT
@@ -34,7 +35,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => NULL::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- reltuples default
 SELECT
@@ -42,7 +44,8 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => NULL::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
 -- relallvisible default
 SELECT
@@ -50,7 +53,17 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => NULL::integer);
+        relallvisible => NULL::integer,
+        relallfrozen => 2::integer);
+
+-- relallfrozen default
+SELECT
+    pg_catalog.pg_set_relation_stats(
+        relation => 'stats_import.test'::regclass,
+        relpages => 17::integer,
+        reltuples => 400.0::real,
+        relallvisible => 4::integer,
+        relallfrozen => NULL::integer);
 
 -- named arguments
 SELECT
@@ -58,9 +71,10 @@ SELECT
         relation => 'stats_import.test'::regclass,
         relpages => 17::integer,
         reltuples => 400.0::real,
-        relallvisible => 4::integer);
+        relallvisible => 4::integer,
+        relallfrozen => 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -94,9 +108,10 @@ SELECT
         'stats_import.test'::regclass,
         18::integer,
         401.0::real,
-        5::integer);
+        5::integer,
+        3::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -109,7 +124,7 @@ SELECT
     pg_catalog.pg_clear_relation_stats(
         'stats_import.test'::regclass);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -589,7 +604,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -597,7 +613,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- reject: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -605,6 +622,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 2::integer,
         'relallvisible');
 
 -- reject: object doesn't exist
@@ -613,7 +631,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
 -- ok: set all stats
 SELECT pg_restore_relation_stats(
@@ -621,9 +640,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -633,7 +653,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '16'::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -643,7 +663,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'reltuples', '500'::real);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -653,7 +673,17 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relallvisible', 5::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -671,9 +701,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -1094,7 +1125,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
 
 -- Generate statistics on table with data
@@ -1249,7 +1281,7 @@ JOIN pg_attribute a ON a.attrelid = s.starelid AND a.attnum = s.staattnum
 WHERE s.starelid = 'stats_import.is_odd'::regclass;
 
 --
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
-- 
2.34.1

#37Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#36)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 05:19:30PM -0500, Melanie Plageman wrote:

Yes, so one thing you haven't said yet is if you are +1 on going
forward with these patches in general.

Sorry, yes, I'm +1 in general. It conceptually makes sense to me that we
should disregard frozen pages when deciding whether to do an insert vacuum,
and it's hard to argue with the results in your original post. I also am
not overly concerned about worker starvation. While this patch does give
higher priority to insert-only/mostly tables, it's also reducing the amount
of resources required to vacuum them, anyway.

--
nathan

#38Robert Treat
rob@xzilla.net
In reply to: Melanie Plageman (#34)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Feb 25, 2025 at 4:29 PM Melanie Plageman
<melanieplageman@gmail.com> wrote:

On Tue, Feb 25, 2025 at 11:36 AM Greg Sabino Mullane <htamfids@gmail.com> wrote:

On Tue, Feb 25, 2025 at 11:03 AM Melanie Plageman <melanieplageman@gmail.com> wrote:

Because users can now manually update these values in pg_class, there wouldn't be a way to detect the difference
between a bogus relallfrozen value due to VM corruption or a bogus value due to manual statistics intervention.

Er..you had me until this. If manual monkeying of the system catalogs leads to a "bogus" error that resembles a real one, then sow the wind, and reap the whirlwind. I don't think that should be a consideration here.

Oh, the WARNING would only show up in a case of actual VM corruption.
The WARNING proposed is after calling visibilitymap_count() before
updating pg_class, if the value we get from the VM for relallfrozen
exceeds relallvisible. If you manually change pg_class
relallfrozen/relallvisible to bogus values, you wouldn't get a
warning. I meant there wouldn't be a way to detect the difference when
viewing pg_class between bogus values because of a corrupt VM and
bogus values because of manual updates to pg_class.

It strikes me as a bit odd to have this extra wording in the pg_class
documentation:

+ Every all-frozen page must also be marked
+       all-visible in the visibility map, so
+       <structfield>relallfrozen</structfield> should be less than or equal to
+       <structfield>relallvisible</structfield>. However, if either field is
+       updated manually or if the visibility map is corrupted, it is possible
+       for <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>.

For example, we don't document that rellallvisible should never exceed
relpages, and we aren't normally in the habit of documenting weird
behavior that might happen if people go updating the system catalogs.
Maybe it's just me, but when I read this earlier, I thought there
might be some intended use case for updating the catalog manually that
you had in mind and so the comments were warranted (and indeed, it's
part of why I thought the warning would be useful for users). But upon
reading the thread more and another pass through your updated patches,
this doesn't seem to be the case, and I wonder if this language might
be more encouraging of people updating catalogs than we would
typically be.

Robert Treat
https://xzilla.net

#39Nathan Bossart
nathandbossart@gmail.com
In reply to: Robert Treat (#38)
Re: Trigger more frequent autovacuums of heavy insert tables

On Wed, Feb 26, 2025 at 11:17:06AM -0500, Robert Treat wrote:

It strikes me as a bit odd to have this extra wording in the pg_class
documentation:

+ Every all-frozen page must also be marked
+       all-visible in the visibility map, so
+       <structfield>relallfrozen</structfield> should be less than or equal to
+       <structfield>relallvisible</structfield>. However, if either field is
+       updated manually or if the visibility map is corrupted, it is possible
+       for <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>.

For example, we don't document that rellallvisible should never exceed
relpages, and we aren't normally in the habit of documenting weird
behavior that might happen if people go updating the system catalogs.
Maybe it's just me, but when I read this earlier, I thought there
might be some intended use case for updating the catalog manually that
you had in mind and so the comments were warranted (and indeed, it's
part of why I thought the warning would be useful for users). But upon
reading the thread more and another pass through your updated patches,
this doesn't seem to be the case, and I wonder if this language might
be more encouraging of people updating catalogs than we would
typically be.

+1. If we did want to add more information about the ordinary expectations
of relallfrozen and friends here, I'd suggest doing so in a separate patch.
IMHO the usual "This is only an estimate..." wording is sufficient for the
introduction of relallfrozen.

--
nathan

#40Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#39)
2 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Wed, Feb 26, 2025 at 12:25 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Wed, Feb 26, 2025 at 11:17:06AM -0500, Robert Treat wrote:

It strikes me as a bit odd to have this extra wording in the pg_class
documentation:

+ Every all-frozen page must also be marked
+       all-visible in the visibility map, so
+       <structfield>relallfrozen</structfield> should be less than or equal to
+       <structfield>relallvisible</structfield>. However, if either field is
+       updated manually or if the visibility map is corrupted, it is possible
+       for <structfield>relallfrozen</structfield> to exceed
+       <structfield>relallvisible</structfield>.

For example, we don't document that rellallvisible should never exceed
relpages, and we aren't normally in the habit of documenting weird
behavior that might happen if people go updating the system catalogs.
Maybe it's just me, but when I read this earlier, I thought there
might be some intended use case for updating the catalog manually that
you had in mind and so the comments were warranted (and indeed, it's
part of why I thought the warning would be useful for users). But upon
reading the thread more and another pass through your updated patches,
this doesn't seem to be the case, and I wonder if this language might
be more encouraging of people updating catalogs than we would
typically be.

+1. If we did want to add more information about the ordinary expectations
of relallfrozen and friends here, I'd suggest doing so in a separate patch.
IMHO the usual "This is only an estimate..." wording is sufficient for the
introduction of relallfrozen.

Makes sense. Thanks Robert and Nathan. Attached v11 changes the docs
wording and is rebased.

- Melanie

Attachments:

v11-0002-Trigger-more-frequent-autovacuums-with-relallfro.patchtext/x-patch; charset=US-ASCII; name=v11-0002-Trigger-more-frequent-autovacuums-with-relallfro.patchDownload
From 957551f8987a090f043587202265210c2406572c Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v11 2/2] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

Reviewed-by: Greg Sabino Mullane
---
 doc/src/sgml/config.sgml                      | 16 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 3 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e55700f35b8..6469e473903 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8773,14 +8773,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ddb303f5201..0aca7d78b90 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5362ff80519..acf45efc4ba 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -675,8 +675,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over active
+						# table size before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

v11-0001-Add-relallfrozen-to-pg_class.patchtext/x-patch; charset=US-ASCII; name=v11-0001-Add-relallfrozen-to-pg_class.patchDownload
From d1aa09c5250ae688e4b7d2102402432b5d5d8dcf Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Mon, 24 Feb 2025 17:27:18 -0500
Subject: [PATCH v11 1/2] Add relallfrozen to pg_class

Add relallfrozen, an estimate of the number of pages marked all-frozen
in the visibility map.

pg_class already has relallvisible, an estimate of the number of pages
in the relation marked all-visible in the visibility map. This is used
primarily for planning.

relallfrozen, however, is useful for estimating the outstanding number
of all-visible but not all-frozen pages in the relation for the purposes
of scheduling manual VACUUMs and tuning vacuum freeze parameters.

In the future, this field could be used to trigger more frequent vacuums
on insert-focused workloads with significant volume of frozen data.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Reviewed-by: Robert Treat <rob@xzilla.net>
Discussion: https://postgr.es/m/flat/Z7ZUWje-e1e_fKeu%40nathan#18ec7b0a585a16a58fe317b4ec42efe0
---
 doc/src/sgml/catalogs.sgml                 |  15 +++
 src/backend/access/heap/vacuumlazy.c       |  18 +++-
 src/backend/catalog/heap.c                 |   2 +
 src/backend/catalog/index.c                |  12 ++-
 src/backend/commands/analyze.c             |  12 +--
 src/backend/commands/cluster.c             |   5 +
 src/backend/commands/vacuum.c              |   6 ++
 src/backend/statistics/relation_stats.c    |  29 +++++-
 src/backend/utils/cache/relcache.c         |   2 +
 src/include/catalog/pg_class.h             |   3 +
 src/include/commands/vacuum.h              |   1 +
 src/test/regress/expected/stats_import.out | 102 +++++++++++++--------
 src/test/regress/sql/stats_import.sql      |  52 +++++++----
 13 files changed, 188 insertions(+), 71 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index ee59a7e15d0..3328dd4478e 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2066,6 +2066,21 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para></entry>
      </row>
 
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>relallfrozen</structfield> <type>int4</type>
+      </para>
+      <para>
+       Number of pages that are marked all-frozen in the table's visibility
+       map.  This is only an estimate used for triggering autovacuums. It is
+       updated by <link linkend="sql-vacuum"><command>VACUUM</command></link>,
+       <link linkend="sql-analyze"><command>ANALYZE</command></link>, and a few
+       DDL commands such as <link linkend="sql-createindex"><command>CREATE
+       INDEX</command></link>.
+      </para></entry>
+     </row>
+
+
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>reltoastrelid</structfield> <type>oid</type>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 1af18a78a2b..3b91d02605a 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -623,7 +623,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 				minmulti_updated;
 	BlockNumber orig_rel_pages,
 				new_rel_pages,
-				new_rel_allvisible;
+				new_rel_allvisible,
+				new_rel_allfrozen;
 	PGRUsage	ru0;
 	TimestampTz starttime = 0;
 	PgStat_Counter startreadtime = 0,
@@ -898,10 +899,18 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * pg_class.relpages to
 	 */
 	new_rel_pages = vacrel->rel_pages;	/* After possible rel truncation */
-	visibilitymap_count(rel, &new_rel_allvisible, NULL);
+	visibilitymap_count(rel, &new_rel_allvisible, &new_rel_allfrozen);
 	if (new_rel_allvisible > new_rel_pages)
 		new_rel_allvisible = new_rel_pages;
 
+	/*
+	 * An all-frozen block _must_ be all-visible. As such, clamp the count of
+	 * all-frozen blocks to the count of all-visible blocks. This matches the
+	 * clamping of relallvisible above.
+	 */
+	if (new_rel_allfrozen > new_rel_allvisible)
+		new_rel_allfrozen = new_rel_allvisible;
+
 	/*
 	 * Now actually update rel's pg_class entry.
 	 *
@@ -910,7 +919,8 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	 * scan every page that isn't skipped using the visibility map.
 	 */
 	vac_update_relstats(rel, new_rel_pages, vacrel->new_live_tuples,
-						new_rel_allvisible, vacrel->nindexes > 0,
+						new_rel_allvisible, new_rel_allfrozen,
+						vacrel->nindexes > 0,
 						vacrel->NewRelfrozenXid, vacrel->NewRelminMxid,
 						&frozenxid_updated, &minmulti_updated, false);
 
@@ -3720,7 +3730,7 @@ update_relstats_all_indexes(LVRelState *vacrel)
 		vac_update_relstats(indrel,
 							istat->num_pages,
 							istat->num_index_tuples,
-							0,
+							0, 0,
 							false,
 							InvalidTransactionId,
 							InvalidMultiXactId,
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 956f196fc95..7ef6f0f1cba 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -924,6 +924,7 @@ InsertPgClassTuple(Relation pg_class_desc,
 	values[Anum_pg_class_relpages - 1] = Int32GetDatum(rd_rel->relpages);
 	values[Anum_pg_class_reltuples - 1] = Float4GetDatum(rd_rel->reltuples);
 	values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
+	values[Anum_pg_class_relallfrozen - 1] = Int32GetDatum(rd_rel->relallfrozen);
 	values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
 	values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
 	values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
@@ -994,6 +995,7 @@ AddNewRelationTuple(Relation pg_class_desc,
 	new_rel_reltup->relpages = 0;
 	new_rel_reltup->reltuples = -1;
 	new_rel_reltup->relallvisible = 0;
+	new_rel_reltup->relallfrozen = 0;
 
 	/* Sequences always have a known size */
 	if (relkind == RELKIND_SEQUENCE)
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index f37b990c81d..8e1741c81f5 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -2793,8 +2793,8 @@ FormIndexDatum(IndexInfo *indexInfo,
  * hasindex: set relhasindex to this value
  * reltuples: if >= 0, set reltuples to this value; else no change
  *
- * If reltuples >= 0, relpages and relallvisible are also updated (using
- * RelationGetNumberOfBlocks() and visibilitymap_count()).
+ * If reltuples >= 0, relpages, relallvisible, and relallfrozen are also
+ * updated (using RelationGetNumberOfBlocks() and visibilitymap_count()).
  *
  * NOTE: an important side-effect of this operation is that an SI invalidation
  * message is sent out to all backends --- including me --- causing relcache
@@ -2812,6 +2812,7 @@ index_update_stats(Relation rel,
 	bool		update_stats;
 	BlockNumber relpages = 0;	/* keep compiler quiet */
 	BlockNumber relallvisible = 0;
+	BlockNumber relallfrozen = 0;
 	Oid			relid = RelationGetRelid(rel);
 	Relation	pg_class;
 	ScanKeyData key[1];
@@ -2851,7 +2852,7 @@ index_update_stats(Relation rel,
 		relpages = RelationGetNumberOfBlocks(rel);
 
 		if (rel->rd_rel->relkind != RELKIND_INDEX)
-			visibilitymap_count(rel, &relallvisible, NULL);
+			visibilitymap_count(rel, &relallvisible, &relallfrozen);
 	}
 
 	/*
@@ -2924,6 +2925,11 @@ index_update_stats(Relation rel,
 			rd_rel->relallvisible = (int32) relallvisible;
 			dirty = true;
 		}
+		if (rd_rel->relallfrozen != (int32) relallfrozen)
+		{
+			rd_rel->relallfrozen = (int32) relallfrozen;
+			dirty = true;
+		}
 	}
 
 	/*
diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c
index cd75954951b..2b5fbdcbd82 100644
--- a/src/backend/commands/analyze.c
+++ b/src/backend/commands/analyze.c
@@ -630,12 +630,11 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 	 */
 	if (!inh)
 	{
-		BlockNumber relallvisible;
+		BlockNumber relallvisible = 0;
+		BlockNumber relallfrozen = 0;
 
 		if (RELKIND_HAS_STORAGE(onerel->rd_rel->relkind))
-			visibilitymap_count(onerel, &relallvisible, NULL);
-		else
-			relallvisible = 0;
+			visibilitymap_count(onerel, &relallvisible, &relallfrozen);
 
 		/*
 		 * Update pg_class for table relation.  CCI first, in case acquirefunc
@@ -646,6 +645,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 							relpages,
 							totalrows,
 							relallvisible,
+							relallfrozen,
 							hasindex,
 							InvalidTransactionId,
 							InvalidMultiXactId,
@@ -662,7 +662,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 			vac_update_relstats(Irel[ind],
 								RelationGetNumberOfBlocks(Irel[ind]),
 								totalindexrows,
-								0,
+								0, 0,
 								false,
 								InvalidTransactionId,
 								InvalidMultiXactId,
@@ -678,7 +678,7 @@ do_analyze_rel(Relation onerel, VacuumParams *params,
 		 */
 		CommandCounterIncrement();
 		vac_update_relstats(onerel, -1, totalrows,
-							0, hasindex, InvalidTransactionId,
+							0, 0, hasindex, InvalidTransactionId,
 							InvalidMultiXactId,
 							NULL, NULL,
 							in_outer_xact);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 99193f5c886..54a08e4102e 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1226,6 +1226,7 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		int32		swap_pages;
 		float4		swap_tuples;
 		int32		swap_allvisible;
+		int32		swap_allfrozen;
 
 		swap_pages = relform1->relpages;
 		relform1->relpages = relform2->relpages;
@@ -1238,6 +1239,10 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
 		swap_allvisible = relform1->relallvisible;
 		relform1->relallvisible = relform2->relallvisible;
 		relform2->relallvisible = swap_allvisible;
+
+		swap_allfrozen = relform1->relallfrozen;
+		relform1->relallfrozen = relform2->relallfrozen;
+		relform2->relallfrozen = swap_allfrozen;
 	}
 
 	/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 0239d9bae65..e81c9a8aba3 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1427,6 +1427,7 @@ void
 vac_update_relstats(Relation relation,
 					BlockNumber num_pages, double num_tuples,
 					BlockNumber num_all_visible_pages,
+					BlockNumber num_all_frozen_pages,
 					bool hasindex, TransactionId frozenxid,
 					MultiXactId minmulti,
 					bool *frozenxid_updated, bool *minmulti_updated,
@@ -1476,6 +1477,11 @@ vac_update_relstats(Relation relation,
 		pgcform->relallvisible = (int32) num_all_visible_pages;
 		dirty = true;
 	}
+	if (pgcform->relallfrozen != (int32) num_all_frozen_pages)
+	{
+		pgcform->relallfrozen = (int32) num_all_frozen_pages;
+		dirty = true;
+	}
 
 	/* Apply DDL updates, but not inside an outer transaction (see above) */
 
diff --git a/src/backend/statistics/relation_stats.c b/src/backend/statistics/relation_stats.c
index 11b1ef2dbc2..2c1cea3fc80 100644
--- a/src/backend/statistics/relation_stats.c
+++ b/src/backend/statistics/relation_stats.c
@@ -36,6 +36,7 @@ enum relation_stats_argnum
 	RELPAGES_ARG,
 	RELTUPLES_ARG,
 	RELALLVISIBLE_ARG,
+	RELALLFROZEN_ARG,
 	NUM_RELATION_STATS_ARGS
 };
 
@@ -45,6 +46,7 @@ static struct StatsArgInfo relarginfo[] =
 	[RELPAGES_ARG] = {"relpages", INT4OID},
 	[RELTUPLES_ARG] = {"reltuples", FLOAT4OID},
 	[RELALLVISIBLE_ARG] = {"relallvisible", INT4OID},
+	[RELALLFROZEN_ARG] = {"relallfrozen", INT4OID},
 	[NUM_RELATION_STATS_ARGS] = {0}
 };
 
@@ -65,11 +67,13 @@ relation_statistics_update(FunctionCallInfo fcinfo)
 	bool		update_reltuples = false;
 	BlockNumber relallvisible = 0;
 	bool		update_relallvisible = false;
+	BlockNumber relallfrozen = 0;
+	bool		update_relallfrozen = false;
 	HeapTuple	ctup;
 	Form_pg_class pgcform;
-	int			replaces[3] = {0};
-	Datum		values[3] = {0};
-	bool		nulls[3] = {0};
+	int			replaces[4] = {0};
+	Datum		values[4] = {0};
+	bool		nulls[4] = {0};
 	int			nreplaces = 0;
 
 	if (!PG_ARGISNULL(RELPAGES_ARG))
@@ -98,6 +102,12 @@ relation_statistics_update(FunctionCallInfo fcinfo)
 		update_relallvisible = true;
 	}
 
+	if (!PG_ARGISNULL(RELALLFROZEN_ARG))
+	{
+		relallfrozen = PG_GETARG_UINT32(RELALLFROZEN_ARG);
+		update_relallfrozen = true;
+	}
+
 	stats_check_required_arg(fcinfo, relarginfo, RELATION_ARG);
 	reloid = PG_GETARG_OID(RELATION_ARG);
 
@@ -148,6 +158,13 @@ relation_statistics_update(FunctionCallInfo fcinfo)
 		nreplaces++;
 	}
 
+	if (update_relallfrozen && relallfrozen != pgcform->relallfrozen)
+	{
+		replaces[nreplaces] = Anum_pg_class_relallfrozen;
+		values[nreplaces] = UInt32GetDatum(relallfrozen);
+		nreplaces++;
+	}
+
 	if (nreplaces > 0)
 	{
 		TupleDesc	tupdesc = RelationGetDescr(crel);
@@ -176,9 +193,9 @@ relation_statistics_update(FunctionCallInfo fcinfo)
 Datum
 pg_clear_relation_stats(PG_FUNCTION_ARGS)
 {
-	LOCAL_FCINFO(newfcinfo, 4);
+	LOCAL_FCINFO(newfcinfo, 5);
 
-	InitFunctionCallInfoData(*newfcinfo, NULL, 4, InvalidOid, NULL, NULL);
+	InitFunctionCallInfoData(*newfcinfo, NULL, 5, InvalidOid, NULL, NULL);
 
 	newfcinfo->args[0].value = PG_GETARG_OID(0);
 	newfcinfo->args[0].isnull = PG_ARGISNULL(0);
@@ -188,6 +205,8 @@ pg_clear_relation_stats(PG_FUNCTION_ARGS)
 	newfcinfo->args[2].isnull = false;
 	newfcinfo->args[3].value = UInt32GetDatum(0);
 	newfcinfo->args[3].isnull = false;
+	newfcinfo->args[4].value = UInt32GetDatum(0);
+	newfcinfo->args[4].isnull = false;
 
 	relation_statistics_update(newfcinfo);
 	PG_RETURN_VOID();
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 398114373e9..d1ae761b3f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1928,6 +1928,7 @@ formrdesc(const char *relationName, Oid relationReltype,
 	relation->rd_rel->relpages = 0;
 	relation->rd_rel->reltuples = -1;
 	relation->rd_rel->relallvisible = 0;
+	relation->rd_rel->relallfrozen = 0;
 	relation->rd_rel->relkind = RELKIND_RELATION;
 	relation->rd_rel->relnatts = (int16) natts;
 
@@ -3885,6 +3886,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
 			classform->relpages = 0;	/* it's empty until further notice */
 			classform->reltuples = -1;
 			classform->relallvisible = 0;
+			classform->relallfrozen = 0;
 		}
 		classform->relfrozenxid = freezeXid;
 		classform->relminmxid = minmulti;
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index f0d612ca487..fa96ba07bf4 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -68,6 +68,9 @@ CATALOG(pg_class,1259,RelationRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83,Relat
 	/* # of all-visible blocks (not always up-to-date) */
 	int32		relallvisible BKI_DEFAULT(0);
 
+	/* # of all-frozen blocks (not always up-to-date) */
+	int32		relallfrozen BKI_DEFAULT(0);
+
 	/* OID of toast table; 0 if none */
 	Oid			reltoastrelid BKI_DEFAULT(0) BKI_LOOKUP_OPT(pg_class);
 
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 1571a66c6bf..baacc63f590 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -349,6 +349,7 @@ extern void vac_update_relstats(Relation relation,
 								BlockNumber num_pages,
 								double num_tuples,
 								BlockNumber num_all_visible_pages,
+								BlockNumber num_all_frozen_pages,
 								bool hasindex,
 								TransactionId frozenxid,
 								MultiXactId minmulti,
diff --git a/src/test/regress/expected/stats_import.out b/src/test/regress/expected/stats_import.out
index 7e8b7f429c9..d48d83f4c47 100644
--- a/src/test/regress/expected/stats_import.out
+++ b/src/test/regress/expected/stats_import.out
@@ -14,12 +14,12 @@ CREATE TABLE stats_import.test(
 ) WITH (autovacuum_enabled = false);
 CREATE INDEX test_i ON stats_import.test(id);
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 BEGIN;
@@ -68,12 +68,12 @@ SELECT
  
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-        0 |        -1 |             0
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+        0 |        -1 |             0 |            0
 (1 row)
 
 --  relpages may be -1 for partitioned tables
@@ -170,18 +170,19 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
  pg_restore_relation_stats 
 ---------------------------
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       17 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       17 |       400 |             4 |            2
 (1 row)
 
 -- ok: just relpages
@@ -194,12 +195,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- ok: just reltuples
@@ -212,12 +213,12 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             4 |            2
 (1 row)
 
 -- ok: just relallvisible
@@ -230,12 +231,30 @@ SELECT pg_restore_relation_stats(
  t
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       500 |             5
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            2
+(1 row)
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+ pg_restore_relation_stats 
+---------------------------
+ t
+(1 row)
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       500 |             5 |            3
 (1 row)
 
 -- warn: bad relpages type
@@ -244,19 +263,20 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 WARNING:  argument "relpages" has type "text", expected type "integer"
  pg_restore_relation_stats 
 ---------------------------
  f
 (1 row)
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
- relpages | reltuples | relallvisible 
-----------+-----------+---------------
-       16 |       400 |             4
+ relpages | reltuples | relallvisible | relallfrozen 
+----------+-----------+---------------+--------------
+       16 |       400 |             4 |            2
 (1 row)
 
 -- invalid relkinds for statistics
@@ -967,7 +987,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
  pg_restore_relation_stats 
 ---------------------------
@@ -1169,7 +1190,8 @@ SELECT pg_catalog.pg_restore_relation_stats(
         'relation', 0::oid,
         'relpages', 17::integer,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 WARNING:  argument "relation" has type "oid", expected type "regclass"
 ERROR:  "relation" cannot be NULL
 --- error: relation not found
@@ -1177,7 +1199,8 @@ SELECT pg_catalog.pg_restore_relation_stats(
         'relation', 0::regclass,
         'relpages', 17::integer,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 ERROR:  could not open relation with OID 0
 -- warn and error: unrecognized argument name
 SELECT pg_restore_relation_stats(
@@ -1185,7 +1208,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'nope', 4::integer);
+        'nope', 4::integer,
+        'relallfrozen', 3::integer);
 WARNING:  unrecognized argument name: "nope"
 ERROR:  could not open relation with OID 0
 -- error: argument name is NULL
@@ -1194,7 +1218,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 ERROR:  name at variadic position 5 is NULL
 -- error: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -1202,7 +1227,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 ERROR:  name at variadic position 5 has type "integer", expected type "text"
 -- error: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -1210,6 +1236,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 3::integer,
         'relallvisible');
 ERROR:  variadic arguments must be name/value pairs
 HINT:  Provide an even number of variadic arguments that can be divided into pairs.
@@ -1219,7 +1246,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 ERROR:  could not open relation with OID 0
 -- error: object does not exist
 SELECT pg_catalog.pg_restore_attribute_stats(
diff --git a/src/test/regress/sql/stats_import.sql b/src/test/regress/sql/stats_import.sql
index 57422750b90..3f515142dda 100644
--- a/src/test/regress/sql/stats_import.sql
+++ b/src/test/regress/sql/stats_import.sql
@@ -18,7 +18,7 @@ CREATE TABLE stats_import.test(
 CREATE INDEX test_i ON stats_import.test(id);
 
 -- starting stats
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -49,7 +49,7 @@ SELECT
     pg_catalog.pg_clear_relation_stats(
         'stats_import.test'::regclass);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -120,9 +120,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -132,7 +133,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '16'::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -142,7 +143,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'reltuples', '500'::real);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -152,7 +153,17 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relallvisible', 5::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
+FROM pg_class
+WHERE oid = 'stats_import.test'::regclass;
+
+-- ok: just relallfrozen
+SELECT pg_restore_relation_stats(
+        'relation', 'stats_import.test'::regclass,
+        'version', 150000::integer,
+        'relallfrozen', 3::integer);
+
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -162,9 +173,10 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', 'nope'::text,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 2::integer);
 
-SELECT relpages, reltuples, relallvisible
+SELECT relpages, reltuples, relallvisible, relallfrozen
 FROM pg_class
 WHERE oid = 'stats_import.test'::regclass;
 
@@ -663,7 +675,8 @@ SELECT * FROM pg_catalog.pg_restore_relation_stats(
     'version', '180000'::integer,
     'relpages', '11'::integer,
     'reltuples', '10000'::real,
-    'relallvisible', '0'::integer
+    'relallvisible', '0'::integer,
+    'relallfrozen', '0'::integer
 );
 
 -- Generate statistics on table with data
@@ -833,14 +846,16 @@ SELECT pg_catalog.pg_restore_relation_stats(
         'relation', 0::oid,
         'relpages', 17::integer,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 
 --- error: relation not found
 SELECT pg_catalog.pg_restore_relation_stats(
         'relation', 0::regclass,
         'relpages', 17::integer,
         'reltuples', 400.0::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 
 -- warn and error: unrecognized argument name
 SELECT pg_restore_relation_stats(
@@ -848,7 +863,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'nope', 4::integer);
+        'nope', 4::integer,
+        'relallfrozen', 3::integer);
 
 -- error: argument name is NULL
 SELECT pg_restore_relation_stats(
@@ -856,7 +872,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         NULL, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 
 -- error: argument name is an integer
 SELECT pg_restore_relation_stats(
@@ -864,7 +881,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         17, '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 
 -- error: odd number of variadic arguments cannot be pairs
 SELECT pg_restore_relation_stats(
@@ -872,6 +890,7 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
+        'relallfrozen', 3::integer,
         'relallvisible');
 
 -- error: object doesn't exist
@@ -880,7 +899,8 @@ SELECT pg_restore_relation_stats(
         'version', 150000::integer,
         'relpages', '17'::integer,
         'reltuples', 400::real,
-        'relallvisible', 4::integer);
+        'relallvisible', 4::integer,
+        'relallfrozen', 3::integer);
 
 -- error: object does not exist
 SELECT pg_catalog.pg_restore_attribute_stats(
-- 
2.34.1

#41Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#40)
Re: Trigger more frequent autovacuums of heavy insert tables

On Wed, Feb 26, 2025 at 04:48:20PM -0500, Melanie Plageman wrote:

Makes sense. Thanks Robert and Nathan. Attached v11 changes the docs
wording and is rebased.

0001 LGTM.

<para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add to
+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table size).
+        This parameter can only be set in the <filename>postgresql.conf</filename>
+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
</para>

nitpick: There might be an unintentional indentation change here.

I'm wondering about the use of the word "active," too. While it's
qualified by the "(unfrozen)" after it, I'm worried it might not be
descriptive enough. For example, I might consider a frozen page that's in
the buffer cache and is being read by queries to be "active." And it
doesn't seem clear to me that it's referring to unfrozen pages and not
unfrozen tuples. Perhaps we should say something like "a fraction of the
unfrozen pages in the table to add...".

+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)

So, if we don't have this data, we just use reltuples, which is the
existing behavior and should trigger vacuums less aggressively than if we
_did_ have the data. That seems like the correct choice to me.

+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);

Makes sense.

--
nathan

#42wenhui qiu
qiuwenhuifx@gmail.com
In reply to: Nathan Bossart (#41)
Re: Trigger more frequent autovacuums of heavy insert tables

Hi

+ * It could be the stats were updated manually and relallfrozen >
+ * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+ * calculations.
+ */
+ relallfrozen = Min(relallfrozen, relpages);
+ pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+ }
+

Based on the comments, the pcnt_unfrozen value could potentially be 0,
which would indicate that everything is frozen. Therefore, is it necessary
to handle the case where the value is 0.?

On Sat, Mar 1, 2025 at 1:54 AM Nathan Bossart <nathandbossart@gmail.com>
wrote:

Show quoted text

On Wed, Feb 26, 2025 at 04:48:20PM -0500, Melanie Plageman wrote:

Makes sense. Thanks Robert and Nathan. Attached v11 changes the docs
wording and is rebased.

0001 LGTM.

<para>
- Specifies a fraction of the table size to add to
- <varname>autovacuum_vacuum_insert_threshold</varname>
- when deciding whether to trigger a <command>VACUUM</command>.
- The default is <literal>0.2</literal> (20% of table size).
- This parameter can only be set in the

<filename>postgresql.conf</filename>

-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+        Specifies a fraction of the active (unfrozen) table size to add

to

+        <varname>autovacuum_vacuum_insert_threshold</varname>
+        when deciding whether to trigger a <command>VACUUM</command>.
+        The default is <literal>0.2</literal> (20% of active table

size).

+ This parameter can only be set in the

<filename>postgresql.conf</filename>

+        file or on the server command line;
+        but the setting can be overridden for individual tables by
+        changing table storage parameters.
</para>

nitpick: There might be an unintentional indentation change here.

I'm wondering about the use of the word "active," too. While it's
qualified by the "(unfrozen)" after it, I'm worried it might not be
descriptive enough. For example, I might consider a frozen page that's in
the buffer cache and is being read by queries to be "active." And it
doesn't seem clear to me that it's referring to unfrozen pages and not
unfrozen tuples. Perhaps we should say something like "a fraction of the
unfrozen pages in the table to add...".

+             /*
+              * If we have data for relallfrozen, calculate the

unfrozen percentage

+ * of the table to modify insert scale factor. This helps

us decide

+ * whether or not to vacuum an insert-heavy table based on

the number

+              * of inserts to the "active" part of the table.
+              */
+             if (relpages > 0 && relallfrozen > 0)

So, if we don't have this data, we just use reltuples, which is the
existing behavior and should trigger vacuums less aggressively than if we
_did_ have the data. That seems like the correct choice to me.

+                     /*
+                      * It could be the stats were updated manually and

relallfrozen >

+ * relpages. Clamp relallfrozen to relpages to

avoid nonsensical

+                      * calculations.
+                      */
+                     relallfrozen = Min(relallfrozen, relpages);
+                     pcnt_unfrozen = 1 - ((float4) relallfrozen /

relpages);

Makes sense.

--
nathan

#43Nathan Bossart
nathandbossart@gmail.com
In reply to: wenhui qiu (#42)
Re: Trigger more frequent autovacuums of heavy insert tables

On Sat, Mar 01, 2025 at 08:57:52AM +0800, wenhui qiu wrote:

Based on the comments, the pcnt_unfrozen value could potentially be 0,
which would indicate that everything is frozen. Therefore, is it necessary
to handle the case where the value is 0.?

How so? If it's 0, then the insert threshold calculation would produce
autovacuum_vacuum_insert_threshold, just like for an empty table. That
seems like the intended behavior to me.

--
nathan

#44Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#41)
1 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Fri, Feb 28, 2025 at 12:54 PM Nathan Bossart
<nathandbossart@gmail.com> wrote:

On Wed, Feb 26, 2025 at 04:48:20PM -0500, Melanie Plageman wrote:

Makes sense. Thanks Robert and Nathan. Attached v11 changes the docs
wording and is rebased.

0001 LGTM.

Cool. Corey checked over the stats import tests off-list and +1'd
them, so I went ahead and pushed this.

<para>
- Specifies a fraction of the table size to add to

<--snip-->

</para>

nitpick: There might be an unintentional indentation change here.

Yep, fixed it in attached v12, thanks.

I'm wondering about the use of the word "active," too. While it's
qualified by the "(unfrozen)" after it, I'm worried it might not be
descriptive enough. For example, I might consider a frozen page that's in
the buffer cache and is being read by queries to be "active." And it
doesn't seem clear to me that it's referring to unfrozen pages and not
unfrozen tuples. Perhaps we should say something like "a fraction of the
unfrozen pages in the table to add...".

Done. Thanks for the suggestion.

I noticed the docs wording is kind of different than that in
postgresql.conf.sample. The docs wording mentions that the scale
factor gets added to the threshold and postgresql.conf.sample does not
(in master as well). I just wanted to make sure my updates to
postgresql.conf.sample sound accurate.

+             /*
+              * If we have data for relallfrozen, calculate the unfrozen percentage
+              * of the table to modify insert scale factor. This helps us decide
+              * whether or not to vacuum an insert-heavy table based on the number
+              * of inserts to the "active" part of the table.
+              */
+             if (relpages > 0 && relallfrozen > 0)

So, if we don't have this data, we just use reltuples, which is the
existing behavior and should trigger vacuums less aggressively than if we
_did_ have the data. That seems like the correct choice to me.

Yep.

- Melanie

Attachments:

v12-0001-Trigger-more-frequent-autovacuums-with-relallfro.patchtext/x-patch; charset=US-ASCII; name=v12-0001-Trigger-more-frequent-autovacuums-with-relallfro.patchDownload
From 33bf7404fc21be611ab9cf8faf0ef593eb9750a0 Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Thu, 16 Jan 2025 16:31:55 -0500
Subject: [PATCH v12] Trigger more frequent autovacuums with relallfrozen

Calculate the insert threshold for triggering an autovacuum of a
relation based on the number of unfrozen pages. By only considering the
"active" (unfrozen) portion of the table when calculating how many
tuples to add to the insert threshold, we can trigger more frequent
vacuums of insert-heavy tables and increase the chances of vacuuming
those pages when they still reside in shared buffers.

This uses the recently added (99f8f3fbbc8f) relallfrozen column of
pg_class.

Reviewed-by: Nathan Bossart <nathandbossart@gmail.com>
Reviewed-by: Greg Sabino Mullane <htamfids@gmail.com>
Discussion: https://postgr.es/m/flat/CAAKRu_aj-P7YyBz_cPNwztz6ohP%2BvWis%3Diz3YcomkB3NpYA--w%40mail.gmail.com
---
 doc/src/sgml/catalogs.sgml                    |  7 ++---
 doc/src/sgml/config.sgml                      | 15 +++++------
 src/backend/postmaster/autovacuum.c           | 27 ++++++++++++++++---
 src/backend/utils/misc/postgresql.conf.sample |  4 +--
 4 files changed, 37 insertions(+), 16 deletions(-)

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 9a21a0d6f15..a3447ad7d4b 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -2072,9 +2072,10 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       </para>
       <para>
        Number of pages that are marked all-frozen in the table's visibility
-       map.  This is only an estimate and can be used along with
-       <structfield>relallvisible</structfield> for scheduling vacuums and
-       tuning <link linkend="runtime-config-vacuum-freezing">vacuum's freezing
+       map.  This is only an estimate used for triggering autovacuums. It can
+       also be used along with <structfield>relallvisible</structfield> for
+       scheduling vacuums and tuning <link
+       linkend="runtime-config-vacuum-freezing">vacuum's freezing
        behavior</link>.
 
        It is updated by
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e55700f35b8..d2fa5f7d1a9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -8773,14 +8773,13 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
        </term>
        <listitem>
         <para>
-         Specifies a fraction of the table size to add to
-         <varname>autovacuum_vacuum_insert_threshold</varname>
-         when deciding whether to trigger a <command>VACUUM</command>.
-         The default is <literal>0.2</literal> (20% of table size).
-         This parameter can only be set in the <filename>postgresql.conf</filename>
-         file or on the server command line;
-         but the setting can be overridden for individual tables by
-         changing table storage parameters.
+         Specifies a fraction of the unfrozen pages in the table to add to
+         <varname>autovacuum_vacuum_insert_threshold</varname> when deciding
+         whether to trigger a <command>VACUUM</command>. The default is
+         <literal>0.2</literal> (20% of unfrozen pages in table). This
+         parameter can only be set in the <filename>postgresql.conf</filename>
+         file or on the server command line; but the setting can be overridden
+         for individual tables by changing table storage parameters.
         </para>
        </listitem>
       </varlistentry>
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ddb303f5201..0aca7d78b90 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -2938,7 +2938,6 @@ relation_needs_vacanalyze(Oid relid,
 {
 	bool		force_vacuum;
 	bool		av_enabled;
-	float4		reltuples;		/* pg_class.reltuples */
 
 	/* constants from reloptions or GUC variables */
 	int			vac_base_thresh,
@@ -3052,7 +3051,11 @@ relation_needs_vacanalyze(Oid relid,
 	 */
 	if (PointerIsValid(tabentry) && AutoVacuumingActive())
 	{
-		reltuples = classForm->reltuples;
+		float4		pcnt_unfrozen = 1;
+		float4		reltuples = classForm->reltuples;
+		int32		relpages = classForm->relpages;
+		int32		relallfrozen = classForm->relallfrozen;
+
 		vactuples = tabentry->dead_tuples;
 		instuples = tabentry->ins_since_vacuum;
 		anltuples = tabentry->mod_since_analyze;
@@ -3061,11 +3064,29 @@ relation_needs_vacanalyze(Oid relid,
 		if (reltuples < 0)
 			reltuples = 0;
 
+		/*
+		 * If we have data for relallfrozen, calculate the unfrozen percentage
+		 * of the table to modify insert scale factor. This helps us decide
+		 * whether or not to vacuum an insert-heavy table based on the number
+		 * of inserts to the "active" part of the table.
+		 */
+		if (relpages > 0 && relallfrozen > 0)
+		{
+			/*
+			 * It could be the stats were updated manually and relallfrozen >
+			 * relpages. Clamp relallfrozen to relpages to avoid nonsensical
+			 * calculations.
+			 */
+			relallfrozen = Min(relallfrozen, relpages);
+			pcnt_unfrozen = 1 - ((float4) relallfrozen / relpages);
+		}
+
 		vacthresh = (float4) vac_base_thresh + vac_scale_factor * reltuples;
 		if (vac_max_thresh >= 0 && vacthresh > (float4) vac_max_thresh)
 			vacthresh = (float4) vac_max_thresh;
 
-		vacinsthresh = (float4) vac_ins_base_thresh + vac_ins_scale_factor * reltuples;
+		vacinsthresh = (float4) vac_ins_base_thresh +
+			vac_ins_scale_factor * reltuples * pcnt_unfrozen;
 		anlthresh = (float4) anl_base_thresh + anl_scale_factor * reltuples;
 
 		/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5362ff80519..8ef9b1a2525 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -675,8 +675,8 @@ autovacuum_worker_slots = 16	# autovacuum worker slots to allocate
 #autovacuum_analyze_threshold = 50	# min number of row updates before
 					# analyze
 #autovacuum_vacuum_scale_factor = 0.2	# fraction of table size before vacuum
-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of unfrozen pages
+            # inserted to before insert vacuum
 #autovacuum_analyze_scale_factor = 0.1	# fraction of table size before analyze
 #autovacuum_vacuum_max_threshold = 100000000    # max number of row updates
 						# before vacuum; -1 disables max
-- 
2.34.1

#45Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#44)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Mar 03, 2025 at 12:18:37PM -0500, Melanie Plageman wrote:

I noticed the docs wording is kind of different than that in
postgresql.conf.sample. The docs wording mentions that the scale
factor gets added to the threshold and postgresql.conf.sample does not
(in master as well). I just wanted to make sure my updates to
postgresql.conf.sample sound accurate.

-#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of inserts over table
-						# size before insert vacuum
+#autovacuum_vacuum_insert_scale_factor = 0.2	# fraction of unfrozen pages
+            # inserted to before insert vacuum

Hm. So for autovacuum_vacuum_scale_factor and
autovacuum_analyze_scale_factor, we have

"fraction of table size before vacuum"
"fraction of table size before analyze"

And currently, for autovacuum_vacuum_insert_scale_factor, we have

"fraction of inserts over table size before insert vacuum"

I think the key change here is that we are replacing "table size" with
"unfrozen pages," which would give us

"fraction of inserts over unfrozen pages before insert vacuum"

However, I find that unclear, if for no other reason than I'm not sure what
"over" means in this context. A reader might think that refers to the
fraction (i.e., inserts / unfrozen pages), or they might think it means
"on" or "above." IMHO your proposed wording is much clearer. The only
part that feels a bit awkward to me is "to before". My mind wants the "to"
to be followed with a verb, so I stumbled the first time I read it and had
to read it again. Perhaps we could just say

"fraction of unfrozen pages before insert vacuum"

That more closely matches the other scale factor parameters. It's
admittedly quite terse, but so are the vast majority of other descriptions
in the sample file. I don't think those are intended to serve as
substitutes for the "real" documentation.

I would make the same argument for leaving out the base threshold. While
it is important information, these extremely concise descriptions don't
really have enough room for the nitty-gritty details.

--
nathan

#46Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#45)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Mar 3, 2025 at 1:11 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

"fraction of unfrozen pages before insert vacuum"

That more closely matches the other scale factor parameters. It's
admittedly quite terse, but so are the vast majority of other descriptions
in the sample file. I don't think those are intended to serve as
substitutes for the "real" documentation.

cool, I've changed it to this, pushed, and marked the commitfest entry
as committed.
Thanks so much for your attention and review!

- Melanie

#47Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#46)
Re: Trigger more frequent autovacuums of heavy insert tables

On Mon, Mar 03, 2025 at 02:47:33PM -0500, Melanie Plageman wrote:

cool, I've changed it to this, pushed, and marked the commitfest entry
as committed.
Thanks so much for your attention and review!

I think maintenance.sgml needs an update (specifically, the part about the
insert threshold [0]https://www.postgresql.org/docs/devel/routine-vacuuming.html#AUTOVACUUM).

[0]: https://www.postgresql.org/docs/devel/routine-vacuuming.html#AUTOVACUUM

--
nathan

#48Nathan Bossart
nathandbossart@gmail.com
In reply to: Nathan Bossart (#47)
1 attachment(s)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Nov 11, 2025 at 12:48:39PM -0600, Nathan Bossart wrote:

I think maintenance.sgml needs an update (specifically, the part about the
insert threshold [0]).

Here is a first try.

--
nathan

Attachments:

v1-0001-fix-insert-vacuum-docs.patchtext/plain; charset=us-asciiDownload
From d5dfbee7894f8ff67bea73be9caa818461fce6b1 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <nathan@postgresql.org>
Date: Tue, 18 Nov 2025 14:17:30 -0600
Subject: [PATCH v1 1/1] fix insert vacuum docs

---
 doc/src/sgml/maintenance.sgml | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml
index 120bac8875f..f4f0433ef6f 100644
--- a/doc/src/sgml/maintenance.sgml
+++ b/doc/src/sgml/maintenance.sgml
@@ -934,12 +934,16 @@ vacuum threshold = Minimum(vacuum max threshold, vacuum base threshold + vacuum
     The table is also vacuumed if the number of tuples inserted since the last
     vacuum has exceeded the defined insert threshold, which is defined as:
 <programlisting>
-vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
+vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples * percent of table not frozen
 </programlisting>
     where the vacuum insert base threshold is
     <xref linkend="guc-autovacuum-vacuum-insert-threshold"/>,
-    and vacuum insert scale factor is
-    <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>.
+    the vacuum insert scale factor is
+    <xref linkend="guc-autovacuum-vacuum-insert-scale-factor"/>,
+    the number of tuples is
+    <structname>pg_class</structname>.<structfield>reltuples</structfield>,
+    and the percent of the table not frozen is
+    <literal>1 - pg_class.relallfrozen / pg_class.relpages</literal>.
     Such vacuums may allow portions of the table to be marked as
     <firstterm>all visible</firstterm> and also allow tuples to be frozen, which
     can reduce the work required in subsequent vacuums.
-- 
2.39.5 (Apple Git-154)

#49Melanie Plageman
melanieplageman@gmail.com
In reply to: Nathan Bossart (#48)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Nov 18, 2025 at 3:20 PM Nathan Bossart <nathandbossart@gmail.com> wrote:

On Tue, Nov 11, 2025 at 12:48:39PM -0600, Nathan Bossart wrote:

I think maintenance.sgml needs an update (specifically, the part about the
insert threshold [0]).

Here is a first try.

Thanks for catching this!

@@ -934,12 +934,16 @@ vacuum threshold = Minimum(vacuum max threshold,
vacuum base threshold + vacuum
     The table is also vacuumed if the number of tuples inserted since the last
     vacuum has exceeded the defined insert threshold, which is defined as:
 <programlisting>
-vacuum insert threshold = vacuum base insert threshold + vacuum
insert scale factor * number of tuples
+vacuum insert threshold = vacuum base insert threshold + vacuum
insert scale factor * number of tuples * percent of table not frozen

I wish we could say "* number of unfrozen tuples". I know that's not
true because we don't know how many tuples are on each page, but the
formula feels a little overly detailed this way. Anyway, this is fine.
I didn't apply and render the whole thing, but the wording looks good
to me.

It's actually interesting that we calculate the thresholds in tuples
when vacuum operates per page. And the per tuple costs are not really
as big of a deal as the per page costs.

- Melanie

#50Nathan Bossart
nathandbossart@gmail.com
In reply to: Melanie Plageman (#49)
Re: Trigger more frequent autovacuums of heavy insert tables

On Tue, Nov 18, 2025 at 04:30:14PM -0500, Melanie Plageman wrote:

I wish we could say "* number of unfrozen tuples". I know that's not
true because we don't know how many tuples are on each page, but the
formula feels a little overly detailed this way. Anyway, this is fine.
I didn't apply and render the whole thing, but the wording looks good
to me.

Committed, thanks for looking.

It's actually interesting that we calculate the thresholds in tuples
when vacuum operates per page. And the per tuple costs are not really
as big of a deal as the per page costs.

Hm... I wonder how much of a difference this makes and whether it's worth
changing.

--
nathan