A few new options for vacuumdb
Hi hackers,
I've attached a few patches to add some options to vacuumdb that might
be useful. Specifically, these patches add the --skip-locked,
--min-xid-age, --min-mxid-age, and --min-relation-size options.
If an option is specified for a server version that is not supported,
the option is silently ignored. For example, SKIP_LOCKED was only
added to VACUUM and ANALYZE for v12. Alternatively, I think we could
fail in vacuum_one_database() if an unsupported option is specified.
Some of these options will work on all currently supported versions,
so I am curious what others think about skipping some of these version
checks altogether.
In this set of patches, I've disallowed using --min-xid-age,
--min-mxid-age, and --min-relation-size in conjunction with the
--table option. IMO this combination of options is prone to
confusion. For example:
vacuumdb mydb --table mytable --min-relation-size 1024
It does not seem clear whether the user wants us to process mytable
only if it is at least 1 GB, or if we should process mytable in
addition to any other relations over 1 GB. Either way, I think trying
to support these combinations of options adds more complexity than it
is worth.
0001 is a minor fix that is somewhat separate from these new options,
although the new options will make the edge case it aims to fix much
easier to reach. When the catalogs are queried in parallel mode to
get the list of tables to process, we currently assume that at least
one table will be returned. If no tables are found, the tables
variable will stay as NULL, which leads to database-wide VACUUM or
ANALYZE commands. Since there are currently no user-configurable
options available for this catalog query, this case is likely
exceptionally rare. However, with the new options, it is much easier
to inadvertently filter out all relations.
I will be adding this work to the next commitfest.
Nathan
Attachments:
v1-0002-Add-skip-locked-option-to-vacuumdb.patchapplication/octet-stream; name=v1-0002-Add-skip-locked-option-to-vacuumdb.patchDownload
From 180ff0bcc0304d093d2e1fc40886b4a0ff4b5d56 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 19 Dec 2018 01:18:02 +0000
Subject: [PATCH v1 2/4] Add --skip-locked option to vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 16 ++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 6 +++++-
src/bin/scripts/vacuumdb.c | 17 +++++++++++++++++
3 files changed, 38 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 955a17a849..942c8ba874 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -167,6 +167,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--skip-locked</option></term>
+ <listitem>
+ <para>
+ Skip relations that cannot be immediately locked for processing.
+ </para>
+ <note>
+ <para>
+ This option is available for servers running PostgreSQL 12 and later.
+ If specified for a server running an older version of PostgreSQL, this
+ option is silently ignored.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-t <replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
<term><option>--table=<replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 4c477a27aa..239ed76834 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 23;
+use Test::More tests => 25;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -33,6 +33,10 @@ $node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
qr/statement: ANALYZE;/,
'vacuumdb -Z');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--skip-locked', 'postgres' ],
+ qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ 'vacuumdb --skip-locked');
$node->command_ok([qw(vacuumdb -Z --table=pg_am dbname=template1)],
'vacuumdb with connection string');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 886895ed2f..5aa2833a73 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -40,6 +40,7 @@ typedef struct vacuumingOptions
bool and_analyze;
bool full;
bool freeze;
+ bool skip_locked;
} vacuumingOptions;
@@ -110,6 +111,7 @@ main(int argc, char *argv[])
{"jobs", required_argument, NULL, 'j'},
{"maintenance-db", required_argument, NULL, 2},
{"analyze-in-stages", no_argument, NULL, 3},
+ {"skip-locked", no_argument, NULL, 4},
{NULL, 0, NULL, 0}
};
@@ -213,6 +215,9 @@ main(int argc, char *argv[])
case 3:
analyze_in_stages = vacopts.analyze_only = true;
break;
+ case 4:
+ vacopts.skip_locked = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -673,6 +678,17 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
appendPQExpBuffer(sql, "%sANALYZE", sep);
sep = comma;
}
+
+ /*
+ * The SKIP_LOCKED option was added for VACUUM and ANALYZE in v12.
+ * Silently ignore it if specified for an older version.
+ */
+ if (vacopts->skip_locked && PQserverVersion(conn) >= 120000)
+ {
+ appendPQExpBuffer(sql, "%sSKIP_LOCKED", sep);
+ sep = comma;
+ }
+
if (sep != paren)
appendPQExpBufferChar(sql, ')');
}
@@ -1011,6 +1027,7 @@ help(const char *progname)
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
+ printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
printf(_(" -v, --verbose write a lot of output\n"));
printf(_(" -V, --version output version information, then exit\n"));
--
2.16.5
v1-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v1-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From 066639fbd1673ad5044ae50ad50af6ff24cd93a3 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 19 Dec 2018 01:18:50 +0000
Subject: [PATCH v1 3/4] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 42 ++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 13 +++++-
src/bin/scripts/vacuumdb.c | 90 +++++++++++++++++++++++++++++++++------
3 files changed, 130 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 942c8ba874..94e8aac268 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -157,6 +157,48 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ </para>
+ <para>
+ Note that this option cannot be used in conjunction with the
+ <option>--table</option> option.
+ </para>
+ <note>
+ <para>
+ This option is available for servers running PostgreSQL 9.5 and later.
+ If specified on a server running an older version of PostgreSQL, this
+ option is silently ignored.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ </para>
+ <para>
+ Note that this option cannot be used in conjunction with the
+ <option>--table</option> option.
+ </para>
+ <note>
+ <para>
+ This option is available for servers running PostgreSQL 7.2 and later.
+ If specified on a server running an older version of PostgreSQL, this
+ option is silently ignored.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 239ed76834..45f79b244e 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 25;
+use Test::More tests => 30;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -64,3 +64,14 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+$node->command_fails(
+ [qw|vacuumdb -Zt funcidx --min-mxid-age 1000000000|],
+ 'specified table with --min-mxid-age option');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=123456789', 'postgres' ],
+ qr/AND greatest\(age\(c.relfrozenxid\), age\(t.relfrozenxid\)\) >= 123456789/,
+ 'vacuumdb --min-xid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-mxid-age=123456789', 'postgres' ],
+ qr/AND greatest\(mxid_age\(c.relminmxid\), mxid_age\(t.relminmxid\)\) >= 123456789/,
+ 'vacuumdb --min-mxid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 5aa2833a73..a2622fab24 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -41,6 +41,8 @@ typedef struct vacuumingOptions
bool full;
bool freeze;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -112,6 +114,8 @@ main(int argc, char *argv[])
{"maintenance-db", required_argument, NULL, 2},
{"analyze-in-stages", no_argument, NULL, 3},
{"skip-locked", no_argument, NULL, 4},
+ {"min-xid-age", required_argument, NULL, 5},
+ {"min-mxid-age", required_argument, NULL, 6},
{NULL, 0, NULL, 0}
};
@@ -218,6 +222,24 @@ main(int argc, char *argv[])
case 4:
vacopts.skip_locked = true;
break;
+ case 5:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 6:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -259,6 +281,13 @@ main(int argc, char *argv[])
/* allow 'and_analyze' with 'analyze_only' */
}
+ if (tables.head != NULL && (vacopts.min_xid_age != 0 || vacopts.min_mxid_age != 0))
+ {
+ fprintf(stderr, _("%s: cannot vacuum specific tables in conjunction with \"%s\" or \"%s\"\n"),
+ progname, "--min-xid-age", "--min-mxid-age");
+ exit(1);
+ }
+
setup_cancel_handler();
/* Avoid opening extra connections. */
@@ -355,6 +384,8 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
int i;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool filter_by_xid_age;
+ bool filter_by_mxid_age;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -383,25 +414,51 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
+ /*
+ * Before 7.2, age() did not exist. Before 9.5, mxid_age() did not exist.
+ */
+ filter_by_xid_age = vacopts->min_xid_age != 0 && PQserverVersion(conn) >= 70200;
+ filter_by_mxid_age = vacopts->min_mxid_age != 0 && PQserverVersion(conn) >= 90500;
+
/*
* If a table list is not provided and we're using multiple connections,
* prepare the list of tables by querying the catalogs.
+ *
+ * If a table list is not provided and we're using one connection, prepare
+ * the list of tables by querying the catalogs only if --min-xid-age or
+ * --min-mxid-age is specified.
*/
- if (parallel && (!tables || !tables->head))
+ if ((parallel && (!tables || !tables->head)) ||
+ (!parallel && (filter_by_xid_age || filter_by_mxid_age)))
{
PQExpBufferData buf;
+ PQExpBufferData catalog_query;
PGresult *res;
int ntups;
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
+ initPQExpBuffer(&catalog_query);
+ appendPQExpBuffer(&catalog_query,
+ "SELECT c.relname, ns.nspname"
+ " FROM pg_class c\n"
+ " JOIN pg_namespace ns ON c.relnamespace = ns.oid\n"
+ " LEFT JOIN pg_class t ON c.reltoastrelid = t.oid\n"
+ " WHERE c.relkind IN ("
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) ")\n");
+
+ if (filter_by_xid_age)
+ appendPQExpBuffer(&catalog_query,
+ " AND greatest(age(c.relfrozenxid), age(t.relfrozenxid)) >= %d\n",
+ vacopts->min_xid_age);
+
+ if (filter_by_mxid_age)
+ appendPQExpBuffer(&catalog_query,
+ " AND greatest(mxid_age(c.relminmxid), mxid_age(t.relminmxid)) >= %d\n",
+ vacopts->min_mxid_age);
+
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
ntups = PQntuples(res);
if (ntups == 0)
@@ -429,10 +486,13 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
* If there are more connections than vacuumable relations, we don't
* need to use them all.
*/
- if (concurrentCons > ntups)
- concurrentCons = ntups;
- if (concurrentCons <= 1)
- parallel = false;
+ if (parallel)
+ {
+ if (concurrentCons > ntups)
+ concurrentCons = ntups;
+ if (concurrentCons <= 1)
+ parallel = false;
+ }
PQclear(res);
}
@@ -1026,6 +1086,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
v1-0004-Add-min-relation-size-option-to-vacuumdb.patchapplication/octet-stream; name=v1-0004-Add-min-relation-size-option-to-vacuumdb.patchDownload
From 8663a1a0c40142764d759bf375f3f7192d7c16e2 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 19 Dec 2018 20:15:39 +0000
Subject: [PATCH v1 4/4] Add --min-relation-size option to vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 21 +++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 9 ++++++++-
src/bin/scripts/vacuumdb.c | 33 ++++++++++++++++++++++++++++++---
3 files changed, 59 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 94e8aac268..0fbccf310e 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -178,6 +178,27 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-relation-size <replaceable class="parameter">size</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a total size
+ of at least <replaceable class="parameter">size</replaceable> megabytes.
+ </para>
+ <para>
+ Note that this option cannot be used in conjunction with the
+ <option>--table</option> option.
+ </para>
+ <note>
+ <para>
+ This option is available for servers running PostgreSQL 8.1 and later.
+ If specified on a server running an older version of PostgreSQL, this
+ option is silently ignored.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
<listitem>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 45f79b244e..982195817e 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 30;
+use Test::More tests => 33;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -75,3 +75,10 @@ $node->issues_sql_like(
[ 'vacuumdb', '--min-mxid-age=123456789', 'postgres' ],
qr/AND greatest\(mxid_age\(c.relminmxid\), mxid_age\(t.relminmxid\)\) >= 123456789/,
'vacuumdb --min-mxid-age');
+$node->command_fails(
+ [qw|vacuumdb -Zt funcidx --min-relation-size 8192|],
+ 'specified table with --min-relation-size option');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-relation-size=8192', 'postgres' ],
+ qr/AND pg_total_relation_size\(c.oid\) >= 8589934592/,
+ 'vacuumdb --min-relation-size');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index a2622fab24..243126bbeb 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,7 @@ typedef struct vacuumingOptions
bool skip_locked;
int min_xid_age;
int min_mxid_age;
+ long min_rel_size_bytes;
} vacuumingOptions;
@@ -116,6 +117,7 @@ main(int argc, char *argv[])
{"skip-locked", no_argument, NULL, 4},
{"min-xid-age", required_argument, NULL, 5},
{"min-mxid-age", required_argument, NULL, 6},
+ {"min-relation-size", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -240,6 +242,15 @@ main(int argc, char *argv[])
exit(1);
}
break;
+ case 7:
+ vacopts.min_rel_size_bytes = (long) atoi(optarg) * 1024L * 1024L;
+ if (vacopts.min_rel_size_bytes <= 0)
+ {
+ fprintf(stderr, _("%s: minimum relation size must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -288,6 +299,13 @@ main(int argc, char *argv[])
exit(1);
}
+ if (tables.head != NULL && vacopts.min_rel_size_bytes != 0)
+ {
+ fprintf(stderr, _("%s: cannot vacuum specific tables in conjunction with \"%s\"\n"),
+ progname, "--min-relation-size");
+ exit(1);
+ }
+
setup_cancel_handler();
/* Avoid opening extra connections. */
@@ -386,6 +404,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
bool parallel = concurrentCons > 1;
bool filter_by_xid_age;
bool filter_by_mxid_age;
+ bool filter_by_rel_size;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -416,20 +435,22 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
/*
* Before 7.2, age() did not exist. Before 9.5, mxid_age() did not exist.
+ * Before 8.1, pg_total_relation_size() did not exist.
*/
filter_by_xid_age = vacopts->min_xid_age != 0 && PQserverVersion(conn) >= 70200;
filter_by_mxid_age = vacopts->min_mxid_age != 0 && PQserverVersion(conn) >= 90500;
+ filter_by_rel_size = vacopts->min_rel_size_bytes != 0 && PQserverVersion(conn) >= 80100;
/*
* If a table list is not provided and we're using multiple connections,
* prepare the list of tables by querying the catalogs.
*
* If a table list is not provided and we're using one connection, prepare
- * the list of tables by querying the catalogs only if --min-xid-age or
- * --min-mxid-age is specified.
+ * the list of tables by querying the catalogs only if --min-xid-age,
+ * --min-mxid-age, or --min-relation-size is specified.
*/
if ((parallel && (!tables || !tables->head)) ||
- (!parallel && (filter_by_xid_age || filter_by_mxid_age)))
+ (!parallel && (filter_by_xid_age || filter_by_mxid_age || filter_by_rel_size)))
{
PQExpBufferData buf;
PQExpBufferData catalog_query;
@@ -456,6 +477,11 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
" AND greatest(mxid_age(c.relminmxid), mxid_age(t.relminmxid)) >= %d\n",
vacopts->min_mxid_age);
+ if (filter_by_rel_size)
+ appendPQExpBuffer(&catalog_query,
+ " AND pg_total_relation_size(c.oid) >= %ld\n",
+ vacopts->min_rel_size_bytes);
+
appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
res = executeQuery(conn, catalog_query.data, progname, echo);
termPQExpBuffer(&catalog_query);
@@ -1087,6 +1113,7 @@ help(const char *progname)
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-relation-size=SIZE miminum size of tables to vacuum, in megabytes\n"));
printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
--
2.16.5
v1-0001-Do-not-process-any-relations-if-the-catalog-query.patchapplication/octet-stream; name=v1-0001-Do-not-process-any-relations-if-the-catalog-query.patchDownload
From bdc89983d9302ab47b94925ac9fe1f5888aeaaa1 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 19 Dec 2018 01:17:25 +0000
Subject: [PATCH v1 1/4] Do not process any relations if the catalog query
returns no results.
---
src/bin/scripts/vacuumdb.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index bcea9e556d..886895ed2f 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -378,8 +378,6 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
-
/*
* If a table list is not provided and we're using multiple connections,
* prepare the list of tables by querying the catalogs.
@@ -390,8 +388,6 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
PGresult *res;
int ntups;
- initPQExpBuffer(&buf);
-
res = executeQuery(conn,
"SELECT c.relname, ns.nspname"
" FROM pg_class c, pg_namespace ns\n"
@@ -403,6 +399,14 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
progname, echo);
ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ initPQExpBuffer(&buf);
for (i = 0; i < ntups; i++)
{
appendPQExpBufferStr(&buf,
@@ -461,6 +465,8 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
+ initPQExpBuffer(&sql);
+
cell = tables ? tables->head : NULL;
do
{
--
2.16.5
On Wed, Dec 19, 2018 at 08:50:10PM +0000, Bossart, Nathan wrote:
If an option is specified for a server version that is not supported,
the option is silently ignored. For example, SKIP_LOCKED was only
added to VACUUM and ANALYZE for v12. Alternatively, I think we could
fail in vacuum_one_database() if an unsupported option is specified.
Some of these options will work on all currently supported versions,
so I am curious what others think about skipping some of these version
checks altogether.
prepare_vacuum_command() already handles that by ignoring silently
unsupported options (see the case of SKIP_LOCKED). So why not doing the
same?
It does not seem clear whether the user wants us to process mytable
only if it is at least 1 GB, or if we should process mytable in
addition to any other relations over 1 GB. Either way, I think trying
to support these combinations of options adds more complexity than it
is worth.
It seems to me that a combination of both options means that the listed
table should be processed only if its minimum size is 1GB. If multiple
tables are specified with --table, then only those reaching 1GB would be
processed. So this restriction can go away. The same applies for the
proposed --min-xid-age and --min-mxid-age.
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable
class="parameter">mxid_age</replaceable>.
+ </para>
Adding a link to explain the multixact business may be helpful, say
vacuum-for-multixact-wraparound. Same comment for XID.
0001 is a minor fix that is somewhat separate from these new options,
although the new options will make the edge case it aims to fix much
easier to reach. When the catalogs are queried in parallel mode to
get the list of tables to process, we currently assume that at least
one table will be returned. If no tables are found, the tables
variable will stay as NULL, which leads to database-wide VACUUM or
ANALYZE commands. Since there are currently no user-configurable
options available for this catalog query, this case is likely
exceptionally rare. However, with the new options, it is much easier
to inadvertently filter out all relations.
Agreed. No need to visibly bother about that in back-branches.
There is an argument about also adding DISABLE_PAGE_SKIPPING.
--
Michael
Hi Michael,
Thanks for taking a look.
On 12/19/18, 8:05 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Wed, Dec 19, 2018 at 08:50:10PM +0000, Bossart, Nathan wrote:
If an option is specified for a server version that is not supported,
the option is silently ignored. For example, SKIP_LOCKED was only
added to VACUUM and ANALYZE for v12. Alternatively, I think we could
fail in vacuum_one_database() if an unsupported option is specified.
Some of these options will work on all currently supported versions,
so I am curious what others think about skipping some of these version
checks altogether.prepare_vacuum_command() already handles that by ignoring silently
unsupported options (see the case of SKIP_LOCKED). So why not doing the
same?
The --skip-locked option in vacuumdb is part of 0002, so I don't think
there's much precedent here. We do currently fall back to the
unparenthesized syntax for VACUUM for versions before 9.0, so there is
some version handling already. What do you think about removing
version checks for unsupported major versions? If we went that route,
these new patches could add version checks only for options that were
added in a supported major version (e.g. mxid_age() was added in 9.5).
Either way, we'll still have to decide whether to fail or to silently
skip the option if you do something like specify --min-mxid-age for a
9.4 server.
It does not seem clear whether the user wants us to process mytable
only if it is at least 1 GB, or if we should process mytable in
addition to any other relations over 1 GB. Either way, I think trying
to support these combinations of options adds more complexity than it
is worth.It seems to me that a combination of both options means that the listed
table should be processed only if its minimum size is 1GB. If multiple
tables are specified with --table, then only those reaching 1GB would be
processed. So this restriction can go away. The same applies for the
proposed --min-xid-age and --min-mxid-age.
Okay. This should be pretty easy to do using individual catalog
lookups before we call prepare_vacuum_command(). Alternatively, I
could add a clause for each specified table in the existing query in
vacuum_one_database(). At that point, it may be cleaner to just use
that catalog query in all cases instead of leaving paths where tables
can remain NULL.
+ <para> + Only execute the vacuum or analyze commands on tables with a multixact + ID age of at least <replaceable class="parameter">mxid_age</replaceable>. + </para> Adding a link to explain the multixact business may be helpful, say vacuum-for-multixact-wraparound. Same comment for XID.
Good idea.
There is an argument about also adding DISABLE_PAGE_SKIPPING.
I'll add this in the next set of patches. I need to add the
parenthesized syntax and --skip-locked for ANALYZE in
prepare_vacuum_command(), too.
Nathan
On Thu, Dec 20, 2018 at 04:48:11PM +0000, Bossart, Nathan wrote:
The --skip-locked option in vacuumdb is part of 0002, so I don't think
there's much precedent here.
It looks like I was not looking at the master branch here ;)
We do currently fall back to the
unparenthesized syntax for VACUUM for versions before 9.0, so there is
some version handling already. What do you think about removing
version checks for unsupported major versions?
I am not exactly sure down to which version this needs to be supported
and what's the policy about that (if others have an opinion to share,
please feel free) but surely it does not hurt to keep those code paths
either from a maintenance point of view.
If we went that route, these new patches could add version checks only
for options that were added in a supported major version (e.g.
mxid_age() was added in 9.5). Either way, we'll still have to decide
whether to fail or to silently skip the option if you do something
like specify --min-mxid-age for a 9.4 server.
There are downsides and upsides for each approach. Reporting a failure
makes it clear that an option is not available with a clear error
message, however it complicates a bit the error handling for parallel
runs. And vacuum_one_database should be the code path doing as that's
where all the connections are taken even when all databases are
processed.
Ignoring the option, as your patch set does, has the disadvantage to
potentially confuse users, say we may see complains like "why is VACUUM
locking even if I specified --skip-locked?". Still this keeps the code
minimalistic and simple.
Just passing down the options and seeing a failure in the query
generated is also a confusing experience I guess for not-so-advanced
users even if it may simplify the error handling.
Issuing a proper error feels more natural to me I think, as users can
react on that properly. Opinions from others are welcome.
--
Michael
On Wed, Dec 19, 2018 at 9:05 PM Michael Paquier <michael@paquier.xyz> wrote:
It does not seem clear whether the user wants us to process mytable
only if it is at least 1 GB, or if we should process mytable in
addition to any other relations over 1 GB. Either way, I think trying
to support these combinations of options adds more complexity than it
is worth.It seems to me that a combination of both options means that the listed
table should be processed only if its minimum size is 1GB. If multiple
tables are specified with --table, then only those reaching 1GB would be
processed.
+1.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan <bossartn@amazon.com> wrote:
Either way, we'll still have to decide whether to fail or to silently
skip the option if you do something like specify --min-mxid-age for a
9.4 server.
+1 for fail.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 12/21/18, 10:51 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:
On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan <bossartn@amazon.com> wrote:
Either way, we'll still have to decide whether to fail or to silently
skip the option if you do something like specify --min-mxid-age for a
9.4 server.+1 for fail.
Sounds good. I'll keep all the version checks and will fail for
unsupported options in the next patch set.
Nathan
On 12/21/18, 11:14 AM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
On 12/21/18, 10:51 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:
On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan <bossartn@amazon.com> wrote:
Either way, we'll still have to decide whether to fail or to silently
skip the option if you do something like specify --min-mxid-age for a
9.4 server.+1 for fail.
Sounds good. I'll keep all the version checks and will fail for
unsupported options in the next patch set.
Here's an updated set of patches with the following changes:
- 0002 adds the parenthesized syntax for ANALYZE.
- 0003 adds DISABLE_PAGE_SKIPPING for VACUUM.
- 0003 also ensures SKIP_LOCKED is applied for --analyze-only.
- 0004 alters vacuumdb to always use the catalog query to discover
the list of tables to process.
- 0003, 0005, and 0006 now fail in vacuum_one_database() if a
specified option is not available on the server version.
- If tables are listed along with --min-xid-age, --min-mxid-age, or
--min-relation-size, each table is processed only if it satisfies
the provided options.
0004 introduces a slight change to existing behavior. Currently, if
you specify a missing table, vacuumdb will process each table until
it reaches the nonexistent one, at which point it will fail. After
0004 is applied, vacuumdb will fail during the catalog query, and no
tables will be processed. I considered avoiding this change in
behavior by doing an additional catalog lookup for each specified
table to see whether it satisfies --min-xid-age, etc. However, this
introduced quite a bit more complexity and increased the number of
catalog queries needed.
Nathan
Attachments:
v2-0006-Add-min-relation-size-option-to-vacuumdb.patchapplication/octet-stream; name=v2-0006-Add-min-relation-size-option-to-vacuumdb.patchDownload
From 2d58da7fd23a6f5f3e27209cb1a06261a3495c0c Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 22:11:32 +0000
Subject: [PATCH v2 6/6] Add --min-relation-size option to vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 16 ++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 6 +++++-
src/bin/scripts/vacuumdb.c | 24 ++++++++++++++++++++++++
3 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index e42d4998ca..c72c245d8d 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -188,6 +188,22 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-relation-size <replaceable class="parameter">size</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a total size
+ of at least <replaceable class="parameter">size</replaceable> megabytes.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 8.1 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
<listitem>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 02683da029..fb608c5180 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 32;
+use Test::More tests => 34;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -80,3 +80,7 @@ $node->issues_sql_like(
[ 'vacuumdb', '--min-xid-age=987654321', 'postgres' ],
qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 987654321/,
'vacuumdb --min-xid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-relation-size=8192', 'postgres' ],
+ qr/AND pg_catalog.pg_total_relation_size\(c.oid\) OPERATOR\(pg_catalog.>=\) 8589934592/,
+ 'vacuumdb --min-relation-size');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 7792fd1c61..dc5f204ceb 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -45,6 +45,7 @@ typedef struct vacuumingOptions
bool skip_locked;
int min_xid_age;
int min_mxid_age;
+ long min_rel_size_bytes;
} vacuumingOptions;
@@ -117,6 +118,7 @@ main(int argc, char *argv[])
{"skip-locked", no_argument, NULL, 5},
{"min-xid-age", required_argument, NULL, 6},
{"min-mxid-age", required_argument, NULL, 7},
+ {"min-relation-size", required_argument, NULL, 8},
{NULL, 0, NULL, 0}
};
@@ -244,6 +246,15 @@ main(int argc, char *argv[])
exit(1);
}
break;
+ case 8:
+ vacopts.min_rel_size_bytes = (long) atoi(optarg) * 1024L * 1024L;
+ if (vacopts.min_rel_size_bytes <= 0)
+ {
+ fprintf(stderr, _("%s: minimum relation size must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -437,6 +448,13 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_rel_size_bytes != 0 && PQserverVersion(conn) < 80100)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 8.1\n"),
+ progname, "--min-relation-size");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -472,6 +490,11 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
" pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
vacopts->min_mxid_age);
+ if (vacopts->min_rel_size_bytes != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND pg_catalog.pg_total_relation_size(c.oid) OPERATOR(pg_catalog.>=) %ld\n",
+ vacopts->min_rel_size_bytes);
+
cell = tables ? tables->head : NULL;
while (cell != NULL)
{
@@ -1112,6 +1135,7 @@ help(const char *progname)
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-relation-size=SIZE minimum size of tables to vacuum, in megabytes\n"));
printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
--
2.16.5
v2-0001-Do-not-process-any-relations-if-the-catalog-query.patchapplication/octet-stream; name=v2-0001-Do-not-process-any-relations-if-the-catalog-query.patchDownload
From 00aa23cb2273063ed33d69a28938b4020b897245 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 20:47:35 +0000
Subject: [PATCH v2 1/6] Do not process any relations if the catalog query
returns no results.
---
src/bin/scripts/vacuumdb.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index e0a8303f3e..7aa5980b51 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -378,8 +378,6 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
-
/*
* If a table list is not provided and we're using multiple connections,
* prepare the list of tables by querying the catalogs.
@@ -390,8 +388,6 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
PGresult *res;
int ntups;
- initPQExpBuffer(&buf);
-
res = executeQuery(conn,
"SELECT c.relname, ns.nspname"
" FROM pg_class c, pg_namespace ns\n"
@@ -403,6 +399,14 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
progname, echo);
ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ initPQExpBuffer(&buf);
for (i = 0; i < ntups; i++)
{
appendPQExpBufferStr(&buf,
@@ -461,6 +465,8 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
+ initPQExpBuffer(&sql);
+
cell = tables ? tables->head : NULL;
do
{
--
2.16.5
v2-0002-Support-parenthesized-syntax-for-ANALYZE-in-vacuu.patchapplication/octet-stream; name=v2-0002-Support-parenthesized-syntax-for-ANALYZE-in-vacuu.patchDownload
From a263b0e325d52f86cbf125318e33df9c5cefeead Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 20:48:13 +0000
Subject: [PATCH v2 2/6] Support parenthesized syntax for ANALYZE in vacuumdb.
---
src/bin/scripts/vacuumdb.c | 78 ++++++++++++++++++++++------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 7aa5980b51..4e2bf713ae 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -639,54 +639,50 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
resetPQExpBuffer(sql);
if (vacopts->analyze_only)
- {
appendPQExpBufferStr(sql, "ANALYZE");
- if (vacopts->verbose)
- appendPQExpBufferStr(sql, " VERBOSE");
- }
else
- {
appendPQExpBufferStr(sql, "VACUUM");
- if (PQserverVersion(conn) >= 90000)
- {
- const char *paren = " (";
- const char *comma = ", ";
- const char *sep = paren;
- if (vacopts->full)
- {
- appendPQExpBuffer(sql, "%sFULL", sep);
- sep = comma;
- }
- if (vacopts->freeze)
- {
- appendPQExpBuffer(sql, "%sFREEZE", sep);
- sep = comma;
- }
- if (vacopts->verbose)
- {
- appendPQExpBuffer(sql, "%sVERBOSE", sep);
- sep = comma;
- }
- if (vacopts->and_analyze)
- {
- appendPQExpBuffer(sql, "%sANALYZE", sep);
- sep = comma;
- }
- if (sep != paren)
- appendPQExpBufferChar(sql, ')');
+ if ((vacopts->analyze_only && PQserverVersion(conn) >= 110000) ||
+ PQserverVersion(conn) >= 90000)
+ {
+ const char *paren = " (";
+ const char *comma = ", ";
+ const char *sep = paren;
+
+ if (vacopts->full)
+ {
+ appendPQExpBuffer(sql, "%sFULL", sep);
+ sep = comma;
}
- else
+ if (vacopts->freeze)
+ {
+ appendPQExpBuffer(sql, "%sFREEZE", sep);
+ sep = comma;
+ }
+ if (vacopts->verbose)
{
- if (vacopts->full)
- appendPQExpBufferStr(sql, " FULL");
- if (vacopts->freeze)
- appendPQExpBufferStr(sql, " FREEZE");
- if (vacopts->verbose)
- appendPQExpBufferStr(sql, " VERBOSE");
- if (vacopts->and_analyze)
- appendPQExpBufferStr(sql, " ANALYZE");
+ appendPQExpBuffer(sql, "%sVERBOSE", sep);
+ sep = comma;
}
+ if (vacopts->and_analyze && !vacopts->analyze_only)
+ {
+ appendPQExpBuffer(sql, "%sANALYZE", sep);
+ sep = comma;
+ }
+ if (sep != paren)
+ appendPQExpBufferChar(sql, ')');
+ }
+ else
+ {
+ if (vacopts->full)
+ appendPQExpBufferStr(sql, " FULL");
+ if (vacopts->freeze)
+ appendPQExpBufferStr(sql, " FREEZE");
+ if (vacopts->verbose)
+ appendPQExpBufferStr(sql, " VERBOSE");
+ if (vacopts->and_analyze && !vacopts->analyze_only)
+ appendPQExpBufferStr(sql, " ANALYZE");
}
if (table)
--
2.16.5
v2-0003-Add-disable-page-skipping-and-skip-locked-to-vacu.patchapplication/octet-stream; name=v2-0003-Add-disable-page-skipping-and-skip-locked-to-vacu.patchDownload
From 6835f259dc90be3f76fda918b55cb5f5622b84b2 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 20:48:40 +0000
Subject: [PATCH v2 3/6] Add --disable-page-skipping and --skip-locked to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 30 ++++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 13 +++++++++++-
src/bin/scripts/vacuumdb.c | 42 +++++++++++++++++++++++++++++++++++++++
3 files changed, 84 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 955a17a849..50b6228f90 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -102,6 +102,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--disable-page-skipping</option></term>
+ <listitem>
+ <para>
+ Disable all page-skipping behavior during processing.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.6 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--echo</option></term>
@@ -167,6 +182,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--skip-locked</option></term>
+ <listitem>
+ <para>
+ Skip relations that cannot be immediately locked for processing.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 12 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-t <replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
<term><option>--table=<replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 4c477a27aa..5f22323006 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 23;
+use Test::More tests => 28;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -33,6 +33,17 @@ $node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
qr/statement: ANALYZE;/,
'vacuumdb -Z');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--disable-page-skipping', 'postgres' ],
+ qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\);/,
+ 'vacuumdb --disable-page-skipping');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--skip-locked', 'postgres' ],
+ qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ 'vacuumdb --skip-locked');
+$node->command_fails(
+ [qw|vacuumdb --analyze-only --disable-page-skipping postgres|],
+ '--analyze-only and --disable-page-skipping specified together');
$node->command_ok([qw(vacuumdb -Z --table=pg_am dbname=template1)],
'vacuumdb with connection string');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 4e2bf713ae..bdc4fcfe35 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -40,6 +40,8 @@ typedef struct vacuumingOptions
bool and_analyze;
bool full;
bool freeze;
+ bool disable_page_skipping;
+ bool skip_locked;
} vacuumingOptions;
@@ -110,6 +112,8 @@ main(int argc, char *argv[])
{"jobs", required_argument, NULL, 'j'},
{"maintenance-db", required_argument, NULL, 2},
{"analyze-in-stages", no_argument, NULL, 3},
+ {"disable-page-skipping", no_argument, NULL, 4},
+ {"skip-locked", no_argument, NULL, 5},
{NULL, 0, NULL, 0}
};
@@ -213,6 +217,12 @@ main(int argc, char *argv[])
case 3:
analyze_in_stages = vacopts.analyze_only = true;
break;
+ case 4:
+ vacopts.disable_page_skipping = true;
+ break;
+ case 5:
+ vacopts.skip_locked = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -251,6 +261,12 @@ main(int argc, char *argv[])
progname, "freeze");
exit(1);
}
+ if (vacopts.disable_page_skipping)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option when performing only analyze\n"),
+ progname, "disable-page-skipping");
+ exit(1);
+ }
/* allow 'and_analyze' with 'analyze_only' */
}
@@ -367,6 +383,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
conn = connectDatabase(dbname, host, port, username, prompt_password,
progname, echo, false, true);
+ if (vacopts->disable_page_skipping && PQserverVersion(conn) < 90600)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.6\n"),
+ progname, "disable-page-skipping");
+ exit(1);
+ }
+
+ if (vacopts->skip_locked && PQserverVersion(conn) < 120000)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 12\n"),
+ progname, "skip-locked");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -670,6 +700,16 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
appendPQExpBuffer(sql, "%sANALYZE", sep);
sep = comma;
}
+ if (vacopts->disable_page_skipping)
+ {
+ appendPQExpBuffer(sql, "%sDISABLE_PAGE_SKIPPING", sep);
+ sep = comma;
+ }
+ if (vacopts->skip_locked)
+ {
+ appendPQExpBuffer(sql, "%sSKIP_LOCKED", sep);
+ sep = comma;
+ }
if (sep != paren)
appendPQExpBufferChar(sql, ')');
}
@@ -1002,11 +1042,13 @@ help(const char *progname)
printf(_("\nOptions:\n"));
printf(_(" -a, --all vacuum all databases\n"));
printf(_(" -d, --dbname=DBNAME database to vacuum\n"));
+ printf(_(" --disable-page-skipping disable all page-skipping behavior\n"));
printf(_(" -e, --echo show the commands being sent to the server\n"));
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
+ printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
printf(_(" -v, --verbose write a lot of output\n"));
printf(_(" -V, --version output version information, then exit\n"));
--
2.16.5
v2-0004-Always-use-a-catalog-query-to-discover-tables-to-.patchapplication/octet-stream; name=v2-0004-Always-use-a-catalog-query-to-discover-tables-to-.patchDownload
From 3c2ee6a10ea529a8eacd4cfb8f6754ca708dfa93 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 22:00:28 +0000
Subject: [PATCH v2 4/6] Always use a catalog query to discover tables to
process in vacuumdb.
---
src/bin/scripts/common.c | 2 +-
src/bin/scripts/common.h | 3 +
src/bin/scripts/t/100_vacuumdb.pl | 12 ++--
src/bin/scripts/vacuumdb.c | 144 +++++++++++++++++++++-----------------
4 files changed, 90 insertions(+), 71 deletions(-)
diff --git a/src/bin/scripts/common.c b/src/bin/scripts/common.c
index 4215bc3d6e..493408d15d 100644
--- a/src/bin/scripts/common.c
+++ b/src/bin/scripts/common.c
@@ -265,7 +265,7 @@ executeMaintenanceCommand(PGconn *conn, const char *query, bool echo)
* finish using them, pg_free(*table). *columns is a pointer into "spec",
* possibly to its NUL terminator.
*/
-static void
+void
split_table_columns_spec(const char *spec, int encoding,
char **table, const char **columns)
{
diff --git a/src/bin/scripts/common.h b/src/bin/scripts/common.h
index 7c3888cefd..f3a7616bae 100644
--- a/src/bin/scripts/common.h
+++ b/src/bin/scripts/common.h
@@ -48,6 +48,9 @@ extern void executeCommand(PGconn *conn, const char *query,
extern bool executeMaintenanceCommand(PGconn *conn, const char *query,
bool echo);
+extern void split_table_columns_spec(const char *spec, int encoding,
+ char **table, const char **columns);
+
extern void appendQualifiedRelation(PQExpBuffer buf, const char *name,
PGconn *conn, const char *progname, bool echo);
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 5f22323006..a9553698ad 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -15,15 +15,15 @@ $node->start;
$node->issues_sql_like(
[ 'vacuumdb', 'postgres' ],
- qr/statement: VACUUM;/,
+ qr/statement: VACUUM pg_catalog\./,
'SQL VACUUM run');
$node->issues_sql_like(
[ 'vacuumdb', '-f', 'postgres' ],
- qr/statement: VACUUM \(FULL\);/,
+ qr/statement: VACUUM \(FULL\) pg_catalog\./,
'vacuumdb -f');
$node->issues_sql_like(
[ 'vacuumdb', '-F', 'postgres' ],
- qr/statement: VACUUM \(FREEZE\);/,
+ qr/statement: VACUUM \(FREEZE\) pg_catalog\./,
'vacuumdb -F');
$node->issues_sql_like(
[ 'vacuumdb', '-zj2', 'postgres' ],
@@ -31,15 +31,15 @@ $node->issues_sql_like(
'vacuumdb -zj2');
$node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
- qr/statement: ANALYZE;/,
+ qr/statement: ANALYZE pg_catalog\./,
'vacuumdb -Z');
$node->issues_sql_like(
[ 'vacuumdb', '--disable-page-skipping', 'postgres' ],
- qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\);/,
+ qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\) pg_catalog\./,
'vacuumdb --disable-page-skipping');
$node->issues_sql_like(
[ 'vacuumdb', '--skip-locked', 'postgres' ],
- qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ qr/statement: VACUUM \(SKIP_LOCKED\) pg_catalog\./,
'vacuumdb --skip-locked');
$node->command_fails(
[qw|vacuumdb --analyze-only --disable-page-skipping postgres|],
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index bdc4fcfe35..78669a35f4 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -19,6 +19,7 @@
#include "catalog/pg_class_d.h"
#include "common.h"
+#include "fe_utils/connect.h"
#include "fe_utils/simple_list.h"
#include "fe_utils/string_utils.h"
@@ -62,9 +63,7 @@ static void vacuum_all_databases(vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet);
static void prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo);
+ vacuumingOptions *vacopts, const char *table);
static void run_vacuum_command(PGconn *conn, const char *sql, bool echo,
const char *table, const char *progname, bool async);
@@ -359,13 +358,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet)
{
PQExpBufferData sql;
+ PQExpBufferData buf;
+ PQExpBufferData catalog_query;
+ PGresult *res;
PGconn *conn;
SimpleStringListCell *cell;
ParallelSlot *slots;
SimpleStringList dbtables = {NULL, NULL};
int i;
+ int ntups;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool first_table = true;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -408,58 +412,84 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- /*
- * If a table list is not provided and we're using multiple connections,
- * prepare the list of tables by querying the catalogs.
- */
- if (parallel && (!tables || !tables->head))
+ initPQExpBuffer(&catalog_query);
+ appendPQExpBuffer(&catalog_query,
+ "SELECT c.relname, ns.nspname"
+ " FROM pg_catalog.pg_class c\n"
+ " JOIN pg_catalog.pg_namespace ns"
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"
+ " WHERE c.relkind IN ("
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) ")\n");
+
+ cell = tables ? tables->head : NULL;
+ while (cell != NULL)
{
- PQExpBufferData buf;
- PGresult *res;
- int ntups;
-
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
-
- ntups = PQntuples(res);
- if (ntups == 0)
+ char *just_table;
+ const char *just_columns;
+
+ split_table_columns_spec(cell->val, PQclientEncoding(conn),
+ &just_table, &just_columns);
+
+ if (first_table)
{
- PQclear(res);
- PQfinish(conn);
- return;
+ appendPQExpBuffer(&catalog_query, " AND (\n c.oid OPERATOR(pg_catalog.=) ");
+ first_table = false;
}
+ else
+ appendPQExpBuffer(&catalog_query, " OR c.oid OPERATOR(pg_catalog.=) ");
- initPQExpBuffer(&buf);
- for (i = 0; i < ntups; i++)
- {
- appendPQExpBufferStr(&buf,
- fmtQualifiedId(PQgetvalue(res, i, 1),
- PQgetvalue(res, i, 0)));
+ appendStringLiteralConn(&catalog_query, just_table, conn);
+ appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n");
- simple_string_list_append(&dbtables, buf.data);
- resetPQExpBuffer(&buf);
- }
+ pg_free(just_table);
- termPQExpBuffer(&buf);
- tables = &dbtables;
+ cell = cell->next;
+ if (cell == NULL)
+ appendPQExpBuffer(&catalog_query, " )\n");
+ }
- /*
- * If there are more connections than vacuumable relations, we don't
- * need to use them all.
- */
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
+
+ ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ initPQExpBuffer(&buf);
+ for (i = 0; i < ntups; i++)
+ {
+ appendPQExpBufferStr(&buf,
+ fmtQualifiedId(PQgetvalue(res, i, 1),
+ PQgetvalue(res, i, 0)));
+
+ simple_string_list_append(&dbtables, buf.data);
+ resetPQExpBuffer(&buf);
+ }
+ termPQExpBuffer(&buf);
+
+ /*
+ * If there are more connections than vacuumable relations, we don't need
+ * to use them all.
+ */
+ if (parallel)
+ {
if (concurrentCons > ntups)
concurrentCons = ntups;
if (concurrentCons <= 1)
parallel = false;
- PQclear(res);
}
+ PQclear(res);
/*
* Setup the database connections. We reuse the connection we already have
@@ -497,10 +527,10 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
initPQExpBuffer(&sql);
- cell = tables ? tables->head : NULL;
+ cell = dbtables.head;
do
{
- const char *tabname = cell ? cell->val : NULL;
+ const char *tabname = cell->val;
ParallelSlot *free_slot;
if (CancelRequested)
@@ -533,12 +563,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
else
free_slot = slots;
- /*
- * Prepare the vacuum command. Note that in some cases this requires
- * query execution, so be sure to use the free connection.
- */
- prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname,
- tables == &dbtables, progname, echo);
+ prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname);
/*
* Execute the vacuum. If not in parallel mode, this terminates the
@@ -548,8 +573,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
run_vacuum_command(free_slot->connection, sql.data,
echo, tabname, progname, parallel);
- if (cell)
- cell = cell->next;
+ cell = cell->next;
} while (cell != NULL);
if (parallel)
@@ -662,9 +686,7 @@ vacuum_all_databases(vacuumingOptions *vacopts,
*/
static void
prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo)
+ vacuumingOptions *vacopts, const char *table)
{
resetPQExpBuffer(sql);
@@ -725,14 +747,8 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
appendPQExpBufferStr(sql, " ANALYZE");
}
- if (table)
- {
- appendPQExpBufferChar(sql, ' ');
- if (table_pre_qualified)
- appendPQExpBufferStr(sql, table);
- else
- appendQualifiedRelation(sql, table, conn, progname, echo);
- }
+ appendPQExpBufferChar(sql, ' ');
+ appendPQExpBufferStr(sql, table);
appendPQExpBufferChar(sql, ';');
}
--
2.16.5
v2-0005-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v2-0005-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From eae3fcb4064de8a7c3ee9d56cb6dc5ab36e9a67a Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Fri, 4 Jan 2019 22:06:28 +0000
Subject: [PATCH v2 5/6] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 32 +++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 11 ++++++++-
src/bin/scripts/vacuumdb.c | 50 +++++++++++++++++++++++++++++++++++++++
3 files changed, 92 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 50b6228f90..e42d4998ca 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,38 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index a9553698ad..02683da029 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 28;
+use Test::More tests => 32;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -71,3 +71,12 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=123456789', 'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 123456789/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=987654321', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 987654321/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 78669a35f4..7792fd1c61 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -401,6 +423,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -424,6 +460,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) ")\n");
+ if (vacopts->min_xid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_xid_age);
+
+ if (vacopts->min_mxid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_mxid_age);
+
cell = tables ? tables->head : NULL;
while (cell != NULL)
{
@@ -1063,6 +1111,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Fri, Jan 04, 2019 at 11:49:46PM +0000, Bossart, Nathan wrote:
0004 introduces a slight change to existing behavior. Currently, if
you specify a missing table, vacuumdb will process each table until
it reaches the nonexistent one, at which point it will fail. After
0004 is applied, vacuumdb will fail during the catalog query, and no
tables will be processed.
I have not looked at the patch set in details, but that would make
vacuumdb more consistent with the way VACUUM works with multiple
relations, which sounds like a good thing.
--
Michael
On Fri, Jan 04, 2019 at 11:49:46PM +0000, Bossart, Nathan wrote:
Here's an updated set of patches with the following changes:
- 0002 adds the parenthesized syntax for ANALYZE.
- 0003 adds DISABLE_PAGE_SKIPPING for VACUUM.
- 0003 also ensures SKIP_LOCKED is applied for --analyze-only.
- 0004 alters vacuumdb to always use the catalog query to discover
the list of tables to process.
- 0003, 0005, and 0006 now fail in vacuum_one_database() if a
specified option is not available on the server version.
- If tables are listed along with --min-xid-age, --min-mxid-age, or
--min-relation-size, each table is processed only if it satisfies
the provided options.
I have been looking at the patch set, and 0001 can actually happen
only once 0005 is applied because this modifies the query doing on
HEAD a full scan of pg_class which would include at least catalog
tables so it can never be empty. For this reason it seems to me that
0001 and 0004 should be merged together as common refactoring, making
sure on the way that all relations exist before processing anything.
As 0005 and 0006 change similar portions of the code, we could also
merge these together in a second patch introducing the new options.
Keeping 0001, 0004, 0005 and 0006 separated eases the review, but I
would likely merge things when they make sense together to reduce
diff chunks.
0002 removes some carefully-written query generation, so as it is
never possible to generate something like ANALYZE FULL. HEAD splits
ANALYZE and VACUUM clearly, but 0002 merges them together so as
careless coding in vacuumdb.c makes it easier to generate command
strings which would fail in the backend relying on the option checks
to make sure that for example combining --full and --analyze-only
never happens. Introducing 0002 is mainly here for 0003, so let's
merge them together.
0004 introduces a slight change to existing behavior. Currently, if
you specify a missing table, vacuumdb will process each table until
it reaches the nonexistent one, at which point it will fail. After
0004 is applied, vacuumdb will fail during the catalog query, and no
tables will be processed. I considered avoiding this change in
behavior by doing an additional catalog lookup for each specified
table to see whether it satisfies --min-xid-age, etc. However, this
introduced quite a bit more complexity and increased the number of
catalog queries needed.
Simplicity is always welcome, knowing that tables are missing before
doing any operations is more consistent with plain VACUUM/ANALYZE.
From patch 0004:
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
We should really avoid that. There are other things like casts which
tend to be forgotten, and if the catalog lookup query gets more
complicated, I feel that this would again be forgotten, reintroducing
the issues that ALWAYS_SECURE_SEARCH_PATH_SQL is here to fix.
I have put my hands into what you sent, and worked on putting
0002/0003 first into shape, finishing with the attached. I have
simplified the query construction a bit, making it IMO easier to read
and to add new options, with comments documenting what's supported
across versions. I have also added a regression test for
--analyze-only --skip-locked. Then I have done some edit of the docs.
What do you think for this portion?
--
Michael
Attachments:
vacuumdb-skip-disable.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index 955a17a849..da4d51e763 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -102,6 +102,23 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--disable-page-skipping</option></term>
+ <listitem>
+ <para>
+ Disable all page-skipping behavior during processing based on
+ the visibility map, similarly to the option
+ <literal>DISABLE_PAGE_SKIPPING</literal> for <command>VACUUM</command>.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running
+ <productname>PostgreSQL</productname> 9.6 and later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-e</option></term>
<term><option>--echo</option></term>
@@ -167,6 +184,21 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--skip-locked</option></term>
+ <listitem>
+ <para>
+ Skip relations that cannot be immediately locked for processing.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running
+ <productname>PostgreSQL</productname> 12 and later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-t <replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
<term><option>--table=<replaceable class="parameter">table</replaceable> [ (<replaceable class="parameter">column</replaceable> [,...]) ]</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 4c477a27aa..4a2c71d665 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 23;
+use Test::More tests => 30;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -33,6 +33,21 @@ $node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
qr/statement: ANALYZE;/,
'vacuumdb -Z');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--disable-page-skipping', 'postgres' ],
+ qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\);/,
+ 'vacuumdb --disable-page-skipping');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--skip-locked', 'postgres' ],
+ qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ 'vacuumdb --skip-locked');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--skip-locked', '--analyze-only', 'postgres' ],
+ qr/statement: ANALYZE \(SKIP_LOCKED\);/,
+ 'vacuumdb --skip-locked --analyze-only');
+$node->command_fails(
+ [qw(vacuumdb --analyze-only --disable-page-skipping postgres)],
+ '--analyze-only and --disable-page-skipping specified together');
$node->command_ok([qw(vacuumdb -Z --table=pg_am dbname=template1)],
'vacuumdb with connection string');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index e0a8303f3e..0b82f5ed50 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -40,6 +40,8 @@ typedef struct vacuumingOptions
bool and_analyze;
bool full;
bool freeze;
+ bool disable_page_skipping;
+ bool skip_locked;
} vacuumingOptions;
@@ -110,6 +112,8 @@ main(int argc, char *argv[])
{"jobs", required_argument, NULL, 'j'},
{"maintenance-db", required_argument, NULL, 2},
{"analyze-in-stages", no_argument, NULL, 3},
+ {"disable-page-skipping", no_argument, NULL, 4},
+ {"skip-locked", no_argument, NULL, 5},
{NULL, 0, NULL, 0}
};
@@ -213,6 +217,12 @@ main(int argc, char *argv[])
case 3:
analyze_in_stages = vacopts.analyze_only = true;
break;
+ case 4:
+ vacopts.disable_page_skipping = true;
+ break;
+ case 5:
+ vacopts.skip_locked = true;
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -251,6 +261,12 @@ main(int argc, char *argv[])
progname, "freeze");
exit(1);
}
+ if (vacopts.disable_page_skipping)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option when performing only analyze\n"),
+ progname, "disable-page-skipping");
+ exit(1);
+ }
/* allow 'and_analyze' with 'analyze_only' */
}
@@ -367,6 +383,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
conn = connectDatabase(dbname, host, port, username, prompt_password,
progname, echo, false, true);
+ if (vacopts->disable_page_skipping && PQserverVersion(conn) < 90600)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.6\n"),
+ progname, "disable-page-skipping");
+ exit(1);
+ }
+
+ if (vacopts->skip_locked && PQserverVersion(conn) < 120000)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 12\n"),
+ progname, "skip-locked");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -630,23 +660,61 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
bool table_pre_qualified,
const char *progname, bool echo)
{
+ const char *paren = " (";
+ const char *comma = ", ";
+ const char *sep = paren;
+
resetPQExpBuffer(sql);
if (vacopts->analyze_only)
{
appendPQExpBufferStr(sql, "ANALYZE");
- if (vacopts->verbose)
- appendPQExpBufferStr(sql, " VERBOSE");
+
+ /* parenthesized grammar of ANALYZE is supported since v11 */
+ if (PQserverVersion(conn) >= 110000)
+ {
+ if (vacopts->skip_locked)
+ {
+ /* SKIP_LOCKED is supported since v12 */
+ Assert(PQserverVersion(conn) >= 120000);
+ appendPQExpBuffer(sql, "%sSKIP_LOCKED", sep);
+ sep = comma;
+ }
+ if (vacopts->verbose)
+ {
+ appendPQExpBuffer(sql, "%sVERBOSE", sep);
+ sep = comma;
+ }
+ if (sep != paren)
+ appendPQExpBufferChar(sql, ')');
+ }
+ else
+ {
+ if (vacopts->verbose)
+ appendPQExpBufferStr(sql, " VERBOSE");
+ }
}
else
{
appendPQExpBufferStr(sql, "VACUUM");
+
+ /* parenthesized grammar of VACUUM is supported since v9.0 */
if (PQserverVersion(conn) >= 90000)
{
- const char *paren = " (";
- const char *comma = ", ";
- const char *sep = paren;
-
+ if (vacopts->disable_page_skipping)
+ {
+ /* DISABLE_PAGE_SKIPPING is supported since v9.6 */
+ Assert(PQserverVersion(conn) >= 90600);
+ appendPQExpBuffer(sql, "%sDISABLE_PAGE_SKIPPING", sep);
+ sep = comma;
+ }
+ if (vacopts->skip_locked)
+ {
+ /* SKIP_LOCKED is supported since v12 */
+ Assert(PQserverVersion(conn) >= 120000);
+ appendPQExpBuffer(sql, "%sSKIP_LOCKED", sep);
+ sep = comma;
+ }
if (vacopts->full)
{
appendPQExpBuffer(sql, "%sFULL", sep);
@@ -1000,11 +1068,13 @@ help(const char *progname)
printf(_("\nOptions:\n"));
printf(_(" -a, --all vacuum all databases\n"));
printf(_(" -d, --dbname=DBNAME database to vacuum\n"));
+ printf(_(" --disable-page-skipping disable all page-skipping behavior\n"));
printf(_(" -e, --echo show the commands being sent to the server\n"));
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
+ printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
printf(_(" -v, --verbose write a lot of output\n"));
printf(_(" -V, --version output version information, then exit\n"));
On Sat, Jan 5, 2019 at 8:50 AM Bossart, Nathan <bossartn@amazon.com> wrote:
On 12/21/18, 11:14 AM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
On 12/21/18, 10:51 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:
On Thu, Dec 20, 2018 at 11:48 AM Bossart, Nathan <bossartn@amazon.com> wrote:
Either way, we'll still have to decide whether to fail or to silently
skip the option if you do something like specify --min-mxid-age for a
9.4 server.+1 for fail.
Sounds good. I'll keep all the version checks and will fail for
unsupported options in the next patch set.Here's an updated set of patches with the following changes:
- 0002 adds the parenthesized syntax for ANALYZE.
- 0003 adds DISABLE_PAGE_SKIPPING for VACUUM.
- 0003 also ensures SKIP_LOCKED is applied for --analyze-only.
- 0004 alters vacuumdb to always use the catalog query to discover
the list of tables to process.
- 0003, 0005, and 0006 now fail in vacuum_one_database() if a
specified option is not available on the server version.
- If tables are listed along with --min-xid-age, --min-mxid-age, or
--min-relation-size, each table is processed only if it satisfies
the provided options.0004 introduces a slight change to existing behavior. Currently, if
you specify a missing table, vacuumdb will process each table until
it reaches the nonexistent one, at which point it will fail. After
0004 is applied, vacuumdb will fail during the catalog query, and no
tables will be processed. I considered avoiding this change in
behavior by doing an additional catalog lookup for each specified
table to see whether it satisfies --min-xid-age, etc. However, this
introduced quite a bit more complexity and increased the number of
catalog queries needed.Nathan
0002 and 0003 are merged and posted by Michael-san and it looks good
to me, so I've looked at the 0001, 0004, 0005 and 0006 patches. Here
is a few review comments.
-----
Even if all tables are filtered by --min-relation-size, min-mxid-age
or min-xid-age the following message appeared.
$ vacuumdb --verbose --min-relation-size 1000000 postgres
vacuumdb: vacuuming database "postgres"
Since no tables are processed in this case isn't is better not to
output this message by moving the message after checking if ntup == 0?
-----
You use pg_total_relation_size() to check the relation size but it
might be better to use pg_relation_size() instead. The
pg_total_relation_size() includes the all index size but I think it's
common based on my experience that we check only the table size
(including toast table) to do vacuum it or not.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Mon, Jan 07, 2019 at 06:10:21PM +0900, Masahiko Sawada wrote:
0002 and 0003 are merged and posted by Michael-san and it looks good
to me, so I've looked at the 0001, 0004, 0005 and 0006 patches. Here
is a few review comments.
I have done another round on 0002/0003 (PQfinish was lacking after
error-ing on incompatible options) and committed the thing.
Even if all tables are filtered by --min-relation-size, min-mxid-age
or min-xid-age the following message appeared.$ vacuumdb --verbose --min-relation-size 1000000 postgres
vacuumdb: vacuuming database "postgres"Since no tables are processed in this case isn't is better not to
output this message by moving the message after checking if ntup ==
0?
Hmm. My take on this one is that we attempt to vacuum relations on
this database, but this may finish by doing nothing. Printing this
message is still important to let the user know that an attempt was
done so I would keep the message.
You use pg_total_relation_size() to check the relation size but it
might be better to use pg_relation_size() instead. The
pg_total_relation_size() includes the all index size but I think it's
common based on my experience that we check only the table size
(including toast table) to do vacuum it or not.
Yes, I am also not completely sure if the way things are defined in
the patch are completely what we are looking for. Still, I have seen
as well people complain about the total amount of space a table is
physically taking on disk, including its indexes. So from the user
experience the direction taken by the patch makes sense, as much as
does the argument you do.
--
Michael
On 1/7/19, 1:12 AM, "Michael Paquier" <michael@paquier.xyz> wrote:
I have been looking at the patch set, and 0001 can actually happen
only once 0005 is applied because this modifies the query doing on
HEAD a full scan of pg_class which would include at least catalog
tables so it can never be empty. For this reason it seems to me that
0001 and 0004 should be merged together as common refactoring, making
sure on the way that all relations exist before processing anything.
As 0005 and 0006 change similar portions of the code, we could also
merge these together in a second patch introducing the new options.
Keeping 0001, 0004, 0005 and 0006 separated eases the review, but I
would likely merge things when they make sense together to reduce
diff chunks.
Thanks for reviewing. Sure, I can merge these together so that it's
just 2 patches.
0002 removes some carefully-written query generation, so as it is
never possible to generate something like ANALYZE FULL. HEAD splits
ANALYZE and VACUUM clearly, but 0002 merges them together so as
careless coding in vacuumdb.c makes it easier to generate command
strings which would fail in the backend relying on the option checks
to make sure that for example combining --full and --analyze-only
never happens. Introducing 0002 is mainly here for 0003, so let's
merge them together.
Makes sense. I was trying to simplify this code a bit, but I agree
with your point about relying on the option checks.
From patch 0004: + executeCommand(conn, "RESET search_path;", progname, echo); + res = executeQuery(conn, catalog_query.data, progname, echo); + termPQExpBuffer(&catalog_query); + PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL, + progname, echo)); We should really avoid that. There are other things like casts which tend to be forgotten, and if the catalog lookup query gets more complicated, I feel that this would again be forgotten, reintroducing the issues that ALWAYS_SECURE_SEARCH_PATH_SQL is here to fix.
This was done in order to maintain the current behavior that
appendQualifiedRelation() gives us. I found that skipping the
search_path handling here forced us to specify the schema in the
argument for --table in most cases. At the very least, I could add a
comment here to highlight the importance of fully qualifying
everything in the catalog query. What do you think?
I have put my hands into what you sent, and worked on putting
0002/0003 first into shape, finishing with the attached. I have
simplified the query construction a bit, making it IMO easier to read
and to add new options, with comments documenting what's supported
across versions. I have also added a regression test for
--analyze-only --skip-locked. Then I have done some edit of the docs.
What do you think for this portion?
Looks good to me, except for one small thing in the documentation:
+ <para>
+ Disable all page-skipping behavior during processing based on
+ the visibility map, similarly to the option
+ <literal>DISABLE_PAGE_SKIPPING</literal> for <command>VACUUM</command>.
+ </para>
I think the "similarly to the option" part is slightly misleading.
It's not just similar, it _is_ using that option in the generated
commands. Perhaps we could point to the VACUUM documentation for more
information about this one.
On 1/7/19, 8:03 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Mon, Jan 07, 2019 at 06:10:21PM +0900, Masahiko Sawada wrote:
0002 and 0003 are merged and posted by Michael-san and it looks good
to me, so I've looked at the 0001, 0004, 0005 and 0006 patches. Here
is a few review comments.I have done another round on 0002/0003 (PQfinish was lacking after
error-ing on incompatible options) and committed the thing.
Thanks!
Even if all tables are filtered by --min-relation-size, min-mxid-age
or min-xid-age the following message appeared.$ vacuumdb --verbose --min-relation-size 1000000 postgres
vacuumdb: vacuuming database "postgres"Since no tables are processed in this case isn't is better not to
output this message by moving the message after checking if ntup ==
0?Hmm. My take on this one is that we attempt to vacuum relations on
this database, but this may finish by doing nothing. Printing this
message is still important to let the user know that an attempt was
done so I would keep the message.
+1 for keeping the message.
You use pg_total_relation_size() to check the relation size but it
might be better to use pg_relation_size() instead. The
pg_total_relation_size() includes the all index size but I think it's
common based on my experience that we check only the table size
(including toast table) to do vacuum it or not.Yes, I am also not completely sure if the way things are defined in
the patch are completely what we are looking for. Still, I have seen
as well people complain about the total amount of space a table is
physically taking on disk, including its indexes. So from the user
experience the direction taken by the patch makes sense, as much as
does the argument you do.
Good point. I think allowing multiple different relation size options
here would be confusing, too (e.g. --min-relation-size versus
--min-total-relation-size). IMO pg_total_relation_size() is the way
to go here, as we'll most likely need to process the indexes and TOAST
tables, too. If/when there is a DISABLE_INDEX_CLEANUP option for
VACUUM, we'd then want to use pg_table_size() when --min-relation-size
and --disable-index-cleanup are used together in vacuumdb. Thoughts?
I've realized that I forgot to add links to the XID/MXID wraparound
documentation like you suggested a while back. I'll make sure that
gets included in the next patch set, too.
Nathan
On Tue, Jan 08, 2019 at 06:46:11PM +0000, Bossart, Nathan wrote:
This was done in order to maintain the current behavior that
appendQualifiedRelation() gives us. I found that skipping the
search_path handling here forced us to specify the schema in the
argument for --table in most cases. At the very least, I could add a
comment here to highlight the importance of fully qualifying
everything in the catalog query. What do you think?
A comment sounds like a good thing. And we really shouldn't hijack
search_path even for one query...
Looks good to me, except for one small thing in the documentation:
+ <para> + Disable all page-skipping behavior during processing based on + the visibility map, similarly to the option + <literal>DISABLE_PAGE_SKIPPING</literal> for <command>VACUUM</command>. + </para>I think the "similarly to the option" part is slightly misleading.
It's not just similar, it _is_ using that option in the generated
commands. Perhaps we could point to the VACUUM documentation for more
information about this one.
Sure. If you have any suggestions, please feel free. Adding a link
to VACUUM documentation sounds good to me, as long as we avoid
duplicating the description related to DISABLE_PAGE_SKIPPING on the
VACUUM page.
Good point. I think allowing multiple different relation size options
here would be confusing, too (e.g. --min-relation-size versus
--min-total-relation-size). IMO pg_total_relation_size() is the way
to go here, as we'll most likely need to process the indexes and TOAST
tables, too. If/when there is a DISABLE_INDEX_CLEANUP option for
VACUUM, we'd then want to use pg_table_size() when --min-relation-size
and --disable-index-cleanup are used together in vacuumdb.
Thoughts?
Yes, using pg_total_relation_size() looks like the best option to me
here as well, still this does not make me 100% comfortable from the
user perspective.
I've realized that I forgot to add links to the XID/MXID wraparound
documentation like you suggested a while back. I'll make sure that
gets included in the next patch set, too.
Nice, thanks.
--
Michael
On Wed, Jan 9, 2019 at 10:06 AM Michael Paquier <michael@paquier.xyz> wrote:
On Tue, Jan 08, 2019 at 06:46:11PM +0000, Bossart, Nathan wrote:
This was done in order to maintain the current behavior that
appendQualifiedRelation() gives us. I found that skipping the
search_path handling here forced us to specify the schema in the
argument for --table in most cases. At the very least, I could add a
comment here to highlight the importance of fully qualifying
everything in the catalog query. What do you think?A comment sounds like a good thing. And we really shouldn't hijack
search_path even for one query...Looks good to me, except for one small thing in the documentation:
+ <para> + Disable all page-skipping behavior during processing based on + the visibility map, similarly to the option + <literal>DISABLE_PAGE_SKIPPING</literal> for <command>VACUUM</command>. + </para>I think the "similarly to the option" part is slightly misleading.
It's not just similar, it _is_ using that option in the generated
commands. Perhaps we could point to the VACUUM documentation for more
information about this one.Sure. If you have any suggestions, please feel free. Adding a link
to VACUUM documentation sounds good to me, as long as we avoid
duplicating the description related to DISABLE_PAGE_SKIPPING on the
VACUUM page.Good point. I think allowing multiple different relation size options
here would be confusing, too (e.g. --min-relation-size versus
--min-total-relation-size). IMO pg_total_relation_size() is the way
to go here, as we'll most likely need to process the indexes and TOAST
tables, too. If/when there is a DISABLE_INDEX_CLEANUP option for
VACUUM, we'd then want to use pg_table_size() when --min-relation-size
and --disable-index-cleanup are used together in vacuumdb.
Thoughts?Yes, using pg_total_relation_size() looks like the best option to me
here as well, still this does not make me 100% comfortable from the
user perspective.
Agreed.
Since pg_(total)_realtion_size() returns 0 for parent table the
specifying the parent table to vacuumdb with --min-relation-size
always does nothing. Maybe we will need to deal with this case when a
function returning whole partitoned table size is introduced.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Wed, Jan 09, 2019 at 10:33:00AM +0900, Masahiko Sawada wrote:
Since pg_(total)_relation_size() returns 0 for parent table the
specifying the parent table to vacuumdb with --min-relation-size
always does nothing. Maybe we will need to deal with this case when a
function returning whole partitoned table size is introduced.
Good point. I am not sure if we want to go down to having a size
function dedicated to partitions especially as this would just now be
a wrapper around pg_partition_tree(), but the size argument with
partitioned tables is something to think about. If we cannot sort out
this part cleanly, perhaps we could just focus on the age-ing
parameters and the other ones first? It seems to me that what is
proposed on this thread has value, so we could shave things and keep
the essential, and focus on what we are sure about for simplicity.
--
Michael
On 1/8/19, 10:34 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Wed, Jan 09, 2019 at 10:33:00AM +0900, Masahiko Sawada wrote:
Since pg_(total)_relation_size() returns 0 for parent table the
specifying the parent table to vacuumdb with --min-relation-size
always does nothing. Maybe we will need to deal with this case when a
function returning whole partitoned table size is introduced.Good point. I am not sure if we want to go down to having a size
function dedicated to partitions especially as this would just now be
a wrapper around pg_partition_tree(), but the size argument with
partitioned tables is something to think about. If we cannot sort out
this part cleanly, perhaps we could just focus on the age-ing
parameters and the other ones first? It seems to me that what is
proposed on this thread has value, so we could shave things and keep
the essential, and focus on what we are sure about for simplicity.
Sounds good. I'll leave out --min-relation-size for now.
Nathan
On Wed, Jan 9, 2019 at 1:33 PM Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Jan 09, 2019 at 10:33:00AM +0900, Masahiko Sawada wrote:
Since pg_(total)_relation_size() returns 0 for parent table the
specifying the parent table to vacuumdb with --min-relation-size
always does nothing. Maybe we will need to deal with this case when a
function returning whole partitoned table size is introduced.Good point. I am not sure if we want to go down to having a size
function dedicated to partitions especially as this would just now be
a wrapper around pg_partition_tree(), but the size argument with
partitioned tables is something to think about. If we cannot sort out
this part cleanly, perhaps we could just focus on the age-ing
parameters and the other ones first? It seems to me that what is
proposed on this thread has value, so we could shave things and keep
the essential, and focus on what we are sure about for simplicity.
Agreed.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
Here's a new patch set that should address the feedback in this
thread. The changes in this version include:
- 0001 is a small fix to the 'vacuumdb --disable-page-skipping'
documentation. My suggestion is to keep it short and simple like
--full, --freeze, --skip-locked, --verbose, and --analyze. The
DISABLE_PAGE_SKIPPING option is well-described in the VACUUM
documentation, and IMO it is reasonably obvious that such vacuumdb
options correspond to the VACUUM options.
- v3-0002 is essentially v2-0001 and v2-0004 combined. I've also
added a comment explaining the importance of fully qualifying the
catalog query used to discover tables to process.
- 0003 includes additional documentation changes explaining the main
uses of --min-xid-age and --min-mxid-age and linking to the
existing wraparound documentation.
As discussed, I've left out the patch for --min-relation-size for now.
Nathan
Attachments:
v3-0001-Adjust-documentation-for-vacuumdb-disable-page-sk.patchapplication/octet-stream; name=v3-0001-Adjust-documentation-for-vacuumdb-disable-page-sk.patchDownload
From 447750567b8a05530cff51773aced191a76694b8 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Sat, 19 Jan 2019 00:33:47 +0000
Subject: [PATCH v3 1/3] Adjust documentation for 'vacuumdb
--disable-page-skipping'.
---
doc/src/sgml/ref/vacuumdb.sgml | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index da4d51e763..f304627802 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -106,9 +106,7 @@ PostgreSQL documentation
<term><option>--disable-page-skipping</option></term>
<listitem>
<para>
- Disable all page-skipping behavior during processing based on
- the visibility map, similarly to the option
- <literal>DISABLE_PAGE_SKIPPING</literal> for <command>VACUUM</command>.
+ Disable skipping pages based on the contents of the visibility map.
</para>
<note>
<para>
--
2.16.5
v3-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchapplication/octet-stream; name=v3-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchDownload
From aed49c21551617cd4fa5c7140256a928d8dd8faf Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Mon, 21 Jan 2019 21:07:07 +0000
Subject: [PATCH v3 2/3] Always use a catalog query to discover tables to
process in vacuumdb.
---
src/bin/scripts/common.c | 2 +-
src/bin/scripts/common.h | 3 +
src/bin/scripts/t/100_vacuumdb.pl | 14 ++--
src/bin/scripts/vacuumdb.c | 148 ++++++++++++++++++++++----------------
4 files changed, 99 insertions(+), 68 deletions(-)
diff --git a/src/bin/scripts/common.c b/src/bin/scripts/common.c
index 4215bc3d6e..493408d15d 100644
--- a/src/bin/scripts/common.c
+++ b/src/bin/scripts/common.c
@@ -265,7 +265,7 @@ executeMaintenanceCommand(PGconn *conn, const char *query, bool echo)
* finish using them, pg_free(*table). *columns is a pointer into "spec",
* possibly to its NUL terminator.
*/
-static void
+void
split_table_columns_spec(const char *spec, int encoding,
char **table, const char **columns)
{
diff --git a/src/bin/scripts/common.h b/src/bin/scripts/common.h
index 7c3888cefd..f3a7616bae 100644
--- a/src/bin/scripts/common.h
+++ b/src/bin/scripts/common.h
@@ -48,6 +48,9 @@ extern void executeCommand(PGconn *conn, const char *query,
extern bool executeMaintenanceCommand(PGconn *conn, const char *query,
bool echo);
+extern void split_table_columns_spec(const char *spec, int encoding,
+ char **table, const char **columns);
+
extern void appendQualifiedRelation(PQExpBuffer buf, const char *name,
PGconn *conn, const char *progname, bool echo);
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 7cb2542e47..3ab54c90e9 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -15,15 +15,15 @@ $node->start;
$node->issues_sql_like(
[ 'vacuumdb', 'postgres' ],
- qr/statement: VACUUM;/,
+ qr/statement: VACUUM pg_catalog\./,
'SQL VACUUM run');
$node->issues_sql_like(
[ 'vacuumdb', '-f', 'postgres' ],
- qr/statement: VACUUM \(FULL\);/,
+ qr/statement: VACUUM \(FULL\) pg_catalog\./,
'vacuumdb -f');
$node->issues_sql_like(
[ 'vacuumdb', '-F', 'postgres' ],
- qr/statement: VACUUM \(FREEZE\);/,
+ qr/statement: VACUUM \(FREEZE\) pg_catalog\./,
'vacuumdb -F');
$node->issues_sql_like(
[ 'vacuumdb', '-zj2', 'postgres' ],
@@ -31,19 +31,19 @@ $node->issues_sql_like(
'vacuumdb -zj2');
$node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
- qr/statement: ANALYZE;/,
+ qr/statement: ANALYZE pg_catalog\./,
'vacuumdb -Z');
$node->issues_sql_like(
[ 'vacuumdb', '--disable-page-skipping', 'postgres' ],
- qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\);/,
+ qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\) pg_catalog\./,
'vacuumdb --disable-page-skipping');
$node->issues_sql_like(
[ 'vacuumdb', '--skip-locked', 'postgres' ],
- qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ qr/statement: VACUUM \(SKIP_LOCKED\) pg_catalog\./,
'vacuumdb --skip-locked');
$node->issues_sql_like(
[ 'vacuumdb', '--skip-locked', '--analyze-only', 'postgres' ],
- qr/statement: ANALYZE \(SKIP_LOCKED\);/,
+ qr/statement: ANALYZE \(SKIP_LOCKED\) pg_catalog\./,
'vacuumdb --skip-locked --analyze-only');
$node->command_fails(
[ 'vacuumdb', '--analyze-only', '--disable-page-skipping', 'postgres' ],
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index ec7d0a326a..490d742125 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -19,6 +19,7 @@
#include "catalog/pg_class_d.h"
#include "common.h"
+#include "fe_utils/connect.h"
#include "fe_utils/simple_list.h"
#include "fe_utils/string_utils.h"
@@ -62,9 +63,7 @@ static void vacuum_all_databases(vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet);
static void prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo);
+ vacuumingOptions *vacopts, const char *table);
static void run_vacuum_command(PGconn *conn, const char *sql, bool echo,
const char *table, const char *progname, bool async);
@@ -359,13 +358,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet)
{
PQExpBufferData sql;
+ PQExpBufferData buf;
+ PQExpBufferData catalog_query;
+ PGresult *res;
PGconn *conn;
SimpleStringListCell *cell;
ParallelSlot *slots;
SimpleStringList dbtables = {NULL, NULL};
int i;
+ int ntups;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool first_table = true;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -410,54 +414,90 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
-
/*
- * If a table list is not provided and we're using multiple connections,
- * prepare the list of tables by querying the catalogs.
+ * Prepare the list of tables to process by querying the catalogs.
+ *
+ * Since we execute the constructed query with the default search_path
+ * (which could be unsafe), it is important to fully qualify everything.
*/
- if (parallel && (!tables || !tables->head))
+ initPQExpBuffer(&catalog_query);
+ appendPQExpBuffer(&catalog_query,
+ "SELECT c.relname, ns.nspname"
+ " FROM pg_catalog.pg_class c\n"
+ " JOIN pg_catalog.pg_namespace ns"
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"
+ " WHERE c.relkind IN ("
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) ")\n");
+
+ cell = tables ? tables->head : NULL;
+ while (cell != NULL)
{
- PQExpBufferData buf;
- PGresult *res;
- int ntups;
-
- initPQExpBuffer(&buf);
-
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
-
- ntups = PQntuples(res);
- for (i = 0; i < ntups; i++)
- {
- appendPQExpBufferStr(&buf,
- fmtQualifiedId(PQgetvalue(res, i, 1),
- PQgetvalue(res, i, 0)));
+ char *just_table;
+ const char *just_columns;
- simple_string_list_append(&dbtables, buf.data);
- resetPQExpBuffer(&buf);
+ split_table_columns_spec(cell->val, PQclientEncoding(conn),
+ &just_table, &just_columns);
+
+ if (first_table)
+ {
+ appendPQExpBuffer(&catalog_query, " AND (\n c.oid OPERATOR(pg_catalog.=) ");
+ first_table = false;
}
+ else
+ appendPQExpBuffer(&catalog_query, " OR c.oid OPERATOR(pg_catalog.=) ");
- termPQExpBuffer(&buf);
- tables = &dbtables;
+ appendStringLiteralConn(&catalog_query, just_table, conn);
+ appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n");
- /*
- * If there are more connections than vacuumable relations, we don't
- * need to use them all.
- */
+ pg_free(just_table);
+
+ cell = cell->next;
+ if (cell == NULL)
+ appendPQExpBuffer(&catalog_query, " )\n");
+ }
+
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
+
+ ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ initPQExpBuffer(&buf);
+ for (i = 0; i < ntups; i++)
+ {
+ appendPQExpBufferStr(&buf,
+ fmtQualifiedId(PQgetvalue(res, i, 1),
+ PQgetvalue(res, i, 0)));
+
+ simple_string_list_append(&dbtables, buf.data);
+ resetPQExpBuffer(&buf);
+ }
+ termPQExpBuffer(&buf);
+
+ /*
+ * If there are more connections than vacuumable relations, we don't need
+ * to use them all.
+ */
+ if (parallel)
+ {
if (concurrentCons > ntups)
concurrentCons = ntups;
if (concurrentCons <= 1)
parallel = false;
- PQclear(res);
}
+ PQclear(res);
/*
* Setup the database connections. We reuse the connection we already have
@@ -493,10 +533,12 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
- cell = tables ? tables->head : NULL;
+ initPQExpBuffer(&sql);
+
+ cell = dbtables.head;
do
{
- const char *tabname = cell ? cell->val : NULL;
+ const char *tabname = cell->val;
ParallelSlot *free_slot;
if (CancelRequested)
@@ -529,12 +571,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
else
free_slot = slots;
- /*
- * Prepare the vacuum command. Note that in some cases this requires
- * query execution, so be sure to use the free connection.
- */
- prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname,
- tables == &dbtables, progname, echo);
+ prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname);
/*
* Execute the vacuum. If not in parallel mode, this terminates the
@@ -544,8 +581,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
run_vacuum_command(free_slot->connection, sql.data,
echo, tabname, progname, parallel);
- if (cell)
- cell = cell->next;
+ cell = cell->next;
} while (cell != NULL);
if (parallel)
@@ -658,9 +694,7 @@ vacuum_all_databases(vacuumingOptions *vacopts,
*/
static void
prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo)
+ vacuumingOptions *vacopts, const char *table)
{
const char *paren = " (";
const char *comma = ", ";
@@ -753,14 +787,8 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
}
}
- if (table)
- {
- appendPQExpBufferChar(sql, ' ');
- if (table_pre_qualified)
- appendPQExpBufferStr(sql, table);
- else
- appendQualifiedRelation(sql, table, conn, progname, echo);
- }
+ appendPQExpBufferChar(sql, ' ');
+ appendPQExpBufferStr(sql, table);
appendPQExpBufferChar(sql, ';');
}
--
2.16.5
v3-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v3-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From 3f59244bb34d2a7869ea000d380d2f2abb28842e Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Sat, 19 Jan 2019 01:20:49 +0000
Subject: [PATCH v3 3/3] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 37 +++++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 11 ++++++++-
src/bin/scripts/vacuumdb.c | 50 +++++++++++++++++++++++++++++++++++++++
3 files changed, 97 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..29ccba2a5a 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,43 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 3ab54c90e9..d5f3f66f5a 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 30;
+use Test::More tests => 34;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -75,3 +75,12 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=123456789', 'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 123456789/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=987654321', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 987654321/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 490d742125..7283ced948 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -403,6 +425,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -432,6 +468,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) ")\n");
+ if (vacopts->min_xid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_xid_age);
+
+ if (vacopts->min_mxid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_mxid_age);
+
cell = tables ? tables->head : NULL;
while (cell != NULL)
{
@@ -1103,6 +1151,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Mon, Jan 21, 2019 at 10:09:09PM +0000, Bossart, Nathan wrote:
Here's a new patch set that should address the feedback in this
thread. The changes in this version include:- 0001 is a small fix to the 'vacuumdb --disable-page-skipping'
documentation. My suggestion is to keep it short and simple like
--full, --freeze, --skip-locked, --verbose, and --analyze. The
DISABLE_PAGE_SKIPPING option is well-described in the VACUUM
documentation, and IMO it is reasonably obvious that such vacuumdb
options correspond to the VACUUM options.
Okay, committed this one.
- v3-0002 is essentially v2-0001 and v2-0004 combined. I've also
added a comment explaining the importance of fully qualifying the
catalog query used to discover tables to process.
Regarding the search_path business, there is actually similar business
in expand_table_name_patterns() for pg_dump if you look close by, so
as users calling --table don't have to specify the schema, and just
stick with patterns.
+ /*
+ * Prepare the list of tables to process by querying the catalogs.
+ *
+ * Since we execute the constructed query with the default search_path
+ * (which could be unsafe), it is important to fully qualify everything.
+ */
It is not only important, but also absolutely mandatory, so let's make
the comment insisting harder on that point. From this point of view,
the query that you are building is visibly correct.
+ appendStringLiteralConn(&catalog_query, just_table, conn);
+ appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n");
+
+ pg_free(just_table);
+
+ cell = cell->next;
+ if (cell == NULL)
+ appendPQExpBuffer(&catalog_query, " )\n");
Hm. It seems to me that you are duplicating what
processSQLNamePattern() does, so we ought to use it. And it is
possible to control the generation of the different WHERE clauses with
a single query based on the number of elements in the list. Perhaps I
am missing something? It looks unnecessary to export
split_table_columns_spec() as well.
- qr/statement: ANALYZE;/,
+ qr/statement: ANALYZE pg_catalog\./,
Or we could just use "ANALYZE \.;/", perhaps patching it first.
Perhaps my regexp here is incorrect, but I just mean to check for an
ANALYZE query which begins by "ANALYZE " and finishes with ";",
without caring about what is in-between. This would make the tests
more portable.
- 0003 includes additional documentation changes explaining the main
uses of --min-xid-age and --min-mxid-age and linking to the
existing wraparound documentation.
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=123456789',
'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\),
pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 123456789/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=987654321', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\),
pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 987654321/,
+ 'vacuumdb --min-xid-age');
It may be better to use numbers which make sure that no relations are
actually fetched, so as if the surrounding tests are messed up we
never make them longer than necessary just to check the shape of the
query.
--
Michael
On 1/21/19, 10:08 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Mon, Jan 21, 2019 at 10:09:09PM +0000, Bossart, Nathan wrote:
Here's a new patch set that should address the feedback in this
thread. The changes in this version include:- 0001 is a small fix to the 'vacuumdb --disable-page-skipping'
documentation. My suggestion is to keep it short and simple like
--full, --freeze, --skip-locked, --verbose, and --analyze. The
DISABLE_PAGE_SKIPPING option is well-described in the VACUUM
documentation, and IMO it is reasonably obvious that such vacuumdb
options correspond to the VACUUM options.Okay, committed this one.
Thanks.
Regarding the search_path business, there is actually similar business
in expand_table_name_patterns() for pg_dump if you look close by, so
as users calling --table don't have to specify the schema, and just
stick with patterns.
I see. The "WHERE c.relkind..." clause looks a bit different than
what I have, so I've altered vacuumdb's catalog query to match. This
was changed in expand_table_name_patterns() as part of 582edc369cd for
a security fix, so we should probably do something similar here.
+ /* + * Prepare the list of tables to process by querying the catalogs. + * + * Since we execute the constructed query with the default search_path + * (which could be unsafe), it is important to fully qualify everything. + */ It is not only important, but also absolutely mandatory, so let's make the comment insisting harder on that point. From this point of view, the query that you are building is visibly correct.
I've updated the comment as suggested.
+ appendStringLiteralConn(&catalog_query, just_table, conn); + appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n"); + + pg_free(just_table); + + cell = cell->next; + if (cell == NULL) + appendPQExpBuffer(&catalog_query, " )\n"); Hm. It seems to me that you are duplicating what processSQLNamePattern() does, so we ought to use it. And it is possible to control the generation of the different WHERE clauses with a single query based on the number of elements in the list. Perhaps I am missing something? It looks unnecessary to export split_table_columns_spec() as well.
I'm sorry, I don't quite follow this. AFAICT processSQLNamePattern()
is for generating WHERE clauses based on a wildcard-pattern string for
psql and pg_dump. This portion of the patch is generating a WHERE
clause to filter only for the tables listed via --table, and I don't
think the --table option currently accepts patterns. Since you can
also provide a list of columns with the --table option, I am using
split_table_columns_spec() to retrieve only the table name for the OID
lookup.
- qr/statement: ANALYZE;/,
+ qr/statement: ANALYZE pg_catalog\./,
Or we could just use "ANALYZE \.;/", perhaps patching it first.
Perhaps my regexp here is incorrect, but I just mean to check for an
ANALYZE query which begins by "ANALYZE " and finishes with ";",
without caring about what is in-between. This would make the tests
more portable.
Sure, I've split this out into a separate patch.
+$node->issues_sql_like( + [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=123456789', 'postgres'], + qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 123456789/, + 'vacuumdb --table --min-mxid-age'); +$node->issues_sql_like( + [ 'vacuumdb', '--min-xid-age=987654321', 'postgres' ], + qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 987654321/, + 'vacuumdb --min-xid-age'); It may be better to use numbers which make sure that no relations are actually fetched, so as if the surrounding tests are messed up we never make them longer than necessary just to check the shape of the query.
I think it's already pretty unlikely that any matching relations would
be found, but I've updated these values to be within the range where
any matching database has already stopped accepting commands to
prevent wraparound.
Nathan
Attachments:
v4-0001-Adjust-vacuumdb-test-regex-in-preparation-for-alw.patchapplication/octet-stream; name=v4-0001-Adjust-vacuumdb-test-regex-in-preparation-for-alw.patchDownload
From 1cdbe18297a1225a1f8665fef4e9b7f4cbf68368 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 22 Jan 2019 22:33:07 +0000
Subject: [PATCH v4 1/3] Adjust vacuumdb test regex in preparation for always
using a catalog query to discover tables.
---
src/bin/scripts/t/100_vacuumdb.pl | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 7cb2542e47..951202b40e 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -15,35 +15,35 @@ $node->start;
$node->issues_sql_like(
[ 'vacuumdb', 'postgres' ],
- qr/statement: VACUUM;/,
+ qr/statement: VACUUM.*;/,
'SQL VACUUM run');
$node->issues_sql_like(
[ 'vacuumdb', '-f', 'postgres' ],
- qr/statement: VACUUM \(FULL\);/,
+ qr/statement: VACUUM \(FULL\).*;/,
'vacuumdb -f');
$node->issues_sql_like(
[ 'vacuumdb', '-F', 'postgres' ],
- qr/statement: VACUUM \(FREEZE\);/,
+ qr/statement: VACUUM \(FREEZE\).*;/,
'vacuumdb -F');
$node->issues_sql_like(
[ 'vacuumdb', '-zj2', 'postgres' ],
- qr/statement: VACUUM \(ANALYZE\) pg_catalog\./,
+ qr/statement: VACUUM \(ANALYZE\).*;/,
'vacuumdb -zj2');
$node->issues_sql_like(
[ 'vacuumdb', '-Z', 'postgres' ],
- qr/statement: ANALYZE;/,
+ qr/statement: ANALYZE.*;/,
'vacuumdb -Z');
$node->issues_sql_like(
[ 'vacuumdb', '--disable-page-skipping', 'postgres' ],
- qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\);/,
+ qr/statement: VACUUM \(DISABLE_PAGE_SKIPPING\).*;/,
'vacuumdb --disable-page-skipping');
$node->issues_sql_like(
[ 'vacuumdb', '--skip-locked', 'postgres' ],
- qr/statement: VACUUM \(SKIP_LOCKED\);/,
+ qr/statement: VACUUM \(SKIP_LOCKED\).*;/,
'vacuumdb --skip-locked');
$node->issues_sql_like(
[ 'vacuumdb', '--skip-locked', '--analyze-only', 'postgres' ],
- qr/statement: ANALYZE \(SKIP_LOCKED\);/,
+ qr/statement: ANALYZE \(SKIP_LOCKED\).*;/,
'vacuumdb --skip-locked --analyze-only');
$node->command_fails(
[ 'vacuumdb', '--analyze-only', '--disable-page-skipping', 'postgres' ],
--
2.16.5
v4-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchapplication/octet-stream; name=v4-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchDownload
From 98803895b2422a4e7ef78b922fb195e116b02110 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 22 Jan 2019 23:08:01 +0000
Subject: [PATCH v4 2/3] Always use a catalog query to discover tables to
process in vacuumdb.
---
src/bin/scripts/common.c | 2 +-
src/bin/scripts/common.h | 3 +
src/bin/scripts/vacuumdb.c | 149 +++++++++++++++++++++++++++------------------
3 files changed, 93 insertions(+), 61 deletions(-)
diff --git a/src/bin/scripts/common.c b/src/bin/scripts/common.c
index 4215bc3d6e..493408d15d 100644
--- a/src/bin/scripts/common.c
+++ b/src/bin/scripts/common.c
@@ -265,7 +265,7 @@ executeMaintenanceCommand(PGconn *conn, const char *query, bool echo)
* finish using them, pg_free(*table). *columns is a pointer into "spec",
* possibly to its NUL terminator.
*/
-static void
+void
split_table_columns_spec(const char *spec, int encoding,
char **table, const char **columns)
{
diff --git a/src/bin/scripts/common.h b/src/bin/scripts/common.h
index 7c3888cefd..f3a7616bae 100644
--- a/src/bin/scripts/common.h
+++ b/src/bin/scripts/common.h
@@ -48,6 +48,9 @@ extern void executeCommand(PGconn *conn, const char *query,
extern bool executeMaintenanceCommand(PGconn *conn, const char *query,
bool echo);
+extern void split_table_columns_spec(const char *spec, int encoding,
+ char **table, const char **columns);
+
extern void appendQualifiedRelation(PQExpBuffer buf, const char *name,
PGconn *conn, const char *progname, bool echo);
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index ec7d0a326a..b72e65c36d 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -19,6 +19,7 @@
#include "catalog/pg_class_d.h"
#include "common.h"
+#include "fe_utils/connect.h"
#include "fe_utils/simple_list.h"
#include "fe_utils/string_utils.h"
@@ -62,9 +63,7 @@ static void vacuum_all_databases(vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet);
static void prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo);
+ vacuumingOptions *vacopts, const char *table);
static void run_vacuum_command(PGconn *conn, const char *sql, bool echo,
const char *table, const char *progname, bool async);
@@ -359,13 +358,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet)
{
PQExpBufferData sql;
+ PQExpBufferData buf;
+ PQExpBufferData catalog_query;
+ PGresult *res;
PGconn *conn;
SimpleStringListCell *cell;
ParallelSlot *slots;
SimpleStringList dbtables = {NULL, NULL};
int i;
+ int ntups;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool first_table = true;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -410,54 +414,91 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
-
/*
- * If a table list is not provided and we're using multiple connections,
- * prepare the list of tables by querying the catalogs.
+ * Prepare the list of tables to process by querying the catalogs.
+ *
+ * Since we execute the constructed query with the default search_path
+ * (which could be unsafe), everything in this query must be fully
+ * qualified.
*/
- if (parallel && (!tables || !tables->head))
+ initPQExpBuffer(&catalog_query);
+ appendPQExpBuffer(&catalog_query,
+ "SELECT c.relname, ns.nspname"
+ " FROM pg_catalog.pg_class c\n"
+ " JOIN pg_catalog.pg_namespace ns"
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"
+ " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) "])\n");
+
+ cell = tables ? tables->head : NULL;
+ while (cell != NULL)
{
- PQExpBufferData buf;
- PGresult *res;
- int ntups;
-
- initPQExpBuffer(&buf);
-
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
-
- ntups = PQntuples(res);
- for (i = 0; i < ntups; i++)
- {
- appendPQExpBufferStr(&buf,
- fmtQualifiedId(PQgetvalue(res, i, 1),
- PQgetvalue(res, i, 0)));
+ char *just_table;
+ const char *just_columns;
- simple_string_list_append(&dbtables, buf.data);
- resetPQExpBuffer(&buf);
+ split_table_columns_spec(cell->val, PQclientEncoding(conn),
+ &just_table, &just_columns);
+
+ if (first_table)
+ {
+ appendPQExpBuffer(&catalog_query, " AND (\n c.oid OPERATOR(pg_catalog.=) ");
+ first_table = false;
}
+ else
+ appendPQExpBuffer(&catalog_query, " OR c.oid OPERATOR(pg_catalog.=) ");
- termPQExpBuffer(&buf);
- tables = &dbtables;
+ appendStringLiteralConn(&catalog_query, just_table, conn);
+ appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n");
- /*
- * If there are more connections than vacuumable relations, we don't
- * need to use them all.
- */
+ pg_free(just_table);
+
+ cell = cell->next;
+ if (cell == NULL)
+ appendPQExpBuffer(&catalog_query, " )\n");
+ }
+
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
+
+ ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ initPQExpBuffer(&buf);
+ for (i = 0; i < ntups; i++)
+ {
+ appendPQExpBufferStr(&buf,
+ fmtQualifiedId(PQgetvalue(res, i, 1),
+ PQgetvalue(res, i, 0)));
+
+ simple_string_list_append(&dbtables, buf.data);
+ resetPQExpBuffer(&buf);
+ }
+ termPQExpBuffer(&buf);
+
+ /*
+ * If there are more connections than vacuumable relations, we don't need
+ * to use them all.
+ */
+ if (parallel)
+ {
if (concurrentCons > ntups)
concurrentCons = ntups;
if (concurrentCons <= 1)
parallel = false;
- PQclear(res);
}
+ PQclear(res);
/*
* Setup the database connections. We reuse the connection we already have
@@ -493,10 +534,12 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
- cell = tables ? tables->head : NULL;
+ initPQExpBuffer(&sql);
+
+ cell = dbtables.head;
do
{
- const char *tabname = cell ? cell->val : NULL;
+ const char *tabname = cell->val;
ParallelSlot *free_slot;
if (CancelRequested)
@@ -529,12 +572,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
else
free_slot = slots;
- /*
- * Prepare the vacuum command. Note that in some cases this requires
- * query execution, so be sure to use the free connection.
- */
- prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname,
- tables == &dbtables, progname, echo);
+ prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname);
/*
* Execute the vacuum. If not in parallel mode, this terminates the
@@ -544,8 +582,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
run_vacuum_command(free_slot->connection, sql.data,
echo, tabname, progname, parallel);
- if (cell)
- cell = cell->next;
+ cell = cell->next;
} while (cell != NULL);
if (parallel)
@@ -658,9 +695,7 @@ vacuum_all_databases(vacuumingOptions *vacopts,
*/
static void
prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo)
+ vacuumingOptions *vacopts, const char *table)
{
const char *paren = " (";
const char *comma = ", ";
@@ -753,14 +788,8 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
}
}
- if (table)
- {
- appendPQExpBufferChar(sql, ' ');
- if (table_pre_qualified)
- appendPQExpBufferStr(sql, table);
- else
- appendQualifiedRelation(sql, table, conn, progname, echo);
- }
+ appendPQExpBufferChar(sql, ' ');
+ appendPQExpBufferStr(sql, table);
appendPQExpBufferChar(sql, ';');
}
--
2.16.5
v4-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v4-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From a97cc74d8511ebb47585b7865caf94a6b44890bd Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 22 Jan 2019 23:11:33 +0000
Subject: [PATCH v4 3/3] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 37 +++++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 11 ++++++++-
src/bin/scripts/vacuumdb.c | 50 +++++++++++++++++++++++++++++++++++++++
3 files changed, 97 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..29ccba2a5a 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,43 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 951202b40e..62355a50e2 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 30;
+use Test::More tests => 34;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -75,3 +75,12 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=2147483000', 'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=2147483001', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 2147483001/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index b72e65c36d..22cf55e8d3 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -403,6 +425,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -433,6 +469,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) "])\n");
+ if (vacopts->min_xid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_xid_age);
+
+ if (vacopts->min_mxid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_mxid_age);
+
cell = tables ? tables->head : NULL;
while (cell != NULL)
{
@@ -1104,6 +1152,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Tue, Jan 22, 2019 at 11:21:32PM +0000, Bossart, Nathan wrote:
On 1/21/19, 10:08 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Mon, Jan 21, 2019 at 10:09:09PM +0000, Bossart, Nathan wrote:
Here's a new patch set that should address the feedback in this
thread. The changes in this version include:- 0001 is a small fix to the 'vacuumdb --disable-page-skipping'
documentation. My suggestion is to keep it short and simple like
--full, --freeze, --skip-locked, --verbose, and --analyze. The
DISABLE_PAGE_SKIPPING option is well-described in the VACUUM
documentation, and IMO it is reasonably obvious that such vacuumdb
options correspond to the VACUUM options.Okay, committed this one.
Thanks.
Regarding the search_path business, there is actually similar business
in expand_table_name_patterns() for pg_dump if you look close by, so
as users calling --table don't have to specify the schema, and just
stick with patterns.I see. The "WHERE c.relkind..." clause looks a bit different than
what I have, so I've altered vacuumdb's catalog query to match. This
was changed in expand_table_name_patterns() as part of 582edc369cd for
a security fix, so we should probably do something similar here.+ /* + * Prepare the list of tables to process by querying the catalogs. + * + * Since we execute the constructed query with the default search_path + * (which could be unsafe), it is important to fully qualify everything. + */ It is not only important, but also absolutely mandatory, so let's make the comment insisting harder on that point. From this point of view, the query that you are building is visibly correct.I've updated the comment as suggested.
+ appendStringLiteralConn(&catalog_query, just_table, conn); + appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass\n"); + + pg_free(just_table); + + cell = cell->next; + if (cell == NULL) + appendPQExpBuffer(&catalog_query, " )\n"); Hm. It seems to me that you are duplicating what processSQLNamePattern() does, so we ought to use it. And it is possible to control the generation of the different WHERE clauses with a single query based on the number of elements in the list. Perhaps I am missing something? It looks unnecessary to export split_table_columns_spec() as well.I'm sorry, I don't quite follow this. AFAICT processSQLNamePattern()
is for generating WHERE clauses based on a wildcard-pattern string for
psql and pg_dump. This portion of the patch is generating a WHERE
clause to filter only for the tables listed via --table, and I don't
think the --table option currently accepts patterns. Since you can
also provide a list of columns with the --table option, I am using
split_table_columns_spec() to retrieve only the table name for the OID
lookup.
Bah, of course you are right. vacuumdb does not support patterns but
I thought it was able to handle that. With column names that would be
sort of funny anyway.
Or we could just use "ANALYZE \.;/", perhaps patching it first.
Perhaps my regexp here is incorrect, but I just mean to check for an
ANALYZE query which begins by "ANALYZE " and finishes with ";",
without caring about what is in-between. This would make the tests
more portable.Sure, I've split this out into a separate patch.
Thanks, I have committed this part to ease the follow-up work.
I think it's already pretty unlikely that any matching relations would
be found, but I've updated these values to be within the range where
any matching database has already stopped accepting commands to
prevent wraparound.
Thanks.
I have been looking at 0002, and I found a problem with the way
ANALYZE queries are generated. For example take this table:
CREATE TABLE aa1 (a int);
Then if I try to run ANALYZE with vacuumdb it just works:
$ vacuumdb -z --table 'aa1(b)'
vacuumdb: vacuuming database "ioltas"
Note that this fails with HEAD, but passes with your patch. The
problem is that the query generated misses the lists of columns when
processing them through split_table_columns_spec(), as what is
generated is that:
VACUUM (ANALYZE) public.aa1;
So the result is actually incorrect because all the columns get
processed.
This patch moves the check about the existence of the relation when
querying the catalogs, perhaps we would want the same for columns for
consistency? Or not. That's a bit harder for sure, not impossible
visibly, still that would mean duplicating a bunch of logic that the
backend is doing by itself, so we could live without it I think.
Attached is a first patch to add more tests in this area, which passes
on HEAD but fails with your patch. It looks like a good idea to tweak
the tests first as there is no coverage yet for the queries generated
with complete and partial column lists.
At the same time, we may want to change split_table_columns_spec's
name to be more consistent with the other APIs, like
splitTableColumnspec. That's a nit though..
--
Michael
Attachments:
vacuumdb-test-cols.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 951202b40e..ff0d97971b 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 30;
+use Test::More tests => 35;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -63,6 +63,7 @@ $node->command_ok(
$node->safe_psql(
'postgres', q|
CREATE TABLE "need""q(uot" (")x" text);
+ CREATE TABLE vactable (a int, b int);
CREATE FUNCTION f0(int) RETURNS int LANGUAGE SQL AS 'SELECT $1 * $1';
CREATE FUNCTION f1(int) RETURNS int LANGUAGE SQL AS 'SELECT f0($1)';
@@ -75,3 +76,15 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+
+$node->command_fails(
+ [ 'vacuumdb', '--analyze', '--table', 'vactable(c)', 'postgres' ],
+ 'incorrect column name with ANALYZE');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--analyze', '--table', 'vactable(a, b)', 'postgres' ],
+ qr/statement: VACUUM \(ANALYZE\) public.vactable\(a, b\);/,
+ 'vacuumdb --analyze with complete column list');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--analyze-only', '--table', 'vactable(b)', 'postgres' ],
+ qr/statement: ANALYZE public.vactable\(b\);/,
+ 'vacuumdb --analyze-only with partial column list');
On 1/22/19, 7:41 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
I have been looking at 0002, and I found a problem with the way
ANALYZE queries are generated. For example take this table:
CREATE TABLE aa1 (a int);Then if I try to run ANALYZE with vacuumdb it just works:
$ vacuumdb -z --table 'aa1(b)'
vacuumdb: vacuuming database "ioltas"Note that this fails with HEAD, but passes with your patch. The
problem is that the query generated misses the lists of columns when
processing them through split_table_columns_spec(), as what is
generated is that:
VACUUM (ANALYZE) public.aa1;So the result is actually incorrect because all the columns get
processed.This patch moves the check about the existence of the relation when
querying the catalogs, perhaps we would want the same for columns for
consistency? Or not. That's a bit harder for sure, not impossible
visibly, still that would mean duplicating a bunch of logic that the
backend is doing by itself, so we could live without it I think.
Oh, wow. Thanks for pointing this out. I should have caught this.
With 0002, we are basically just throwing out the column lists
entirely as we obtain the qualified identifiers from the catalog
query. To fix this, I've added an optional CTE for tracking any
provided column lists. v5-0001 is your test patch for this case, and
v5-0002 splits out the work for split_table_columns_spec().
Nathan
Attachments:
v5-0001-Add-tests-for-using-vacuumdb-with-column-lists-pr.patchapplication/octet-stream; name=v5-0001-Add-tests-for-using-vacuumdb-with-column-lists-pr.patchDownload
From 33305f7506be52ad1926d34d4413db9d4cd48472 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 23 Jan 2019 22:07:01 +0000
Subject: [PATCH v5 1/4] Add tests for using vacuumdb with column lists
provided.
---
src/bin/scripts/t/100_vacuumdb.pl | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 951202b40e..ff0d97971b 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 30;
+use Test::More tests => 35;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -63,6 +63,7 @@ $node->command_ok(
$node->safe_psql(
'postgres', q|
CREATE TABLE "need""q(uot" (")x" text);
+ CREATE TABLE vactable (a int, b int);
CREATE FUNCTION f0(int) RETURNS int LANGUAGE SQL AS 'SELECT $1 * $1';
CREATE FUNCTION f1(int) RETURNS int LANGUAGE SQL AS 'SELECT f0($1)';
@@ -75,3 +76,15 @@ $node->command_ok([qw|vacuumdb -Z --table="need""q(uot"(")x") postgres|],
$node->command_fails(
[qw|vacuumdb -Zt funcidx postgres|],
'unqualifed name via functional index');
+
+$node->command_fails(
+ [ 'vacuumdb', '--analyze', '--table', 'vactable(c)', 'postgres' ],
+ 'incorrect column name with ANALYZE');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--analyze', '--table', 'vactable(a, b)', 'postgres' ],
+ qr/statement: VACUUM \(ANALYZE\) public.vactable\(a, b\);/,
+ 'vacuumdb --analyze with complete column list');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--analyze-only', '--table', 'vactable(b)', 'postgres' ],
+ qr/statement: ANALYZE public.vactable\(b\);/,
+ 'vacuumdb --analyze-only with partial column list');
--
2.16.5
v5-0002-Export-split_table_columns_spec-for-use-in-vacuum.patchapplication/octet-stream; name=v5-0002-Export-split_table_columns_spec-for-use-in-vacuum.patchDownload
From 0686bfafda890148428fbcff215db932c54c0ade Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 23 Jan 2019 22:18:56 +0000
Subject: [PATCH v5 2/4] Export split_table_columns_spec() for use in vacuumdb.
---
src/bin/scripts/common.c | 8 ++++----
src/bin/scripts/common.h | 3 +++
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/src/bin/scripts/common.c b/src/bin/scripts/common.c
index 4215bc3d6e..7139b7c667 100644
--- a/src/bin/scripts/common.c
+++ b/src/bin/scripts/common.c
@@ -265,9 +265,9 @@ executeMaintenanceCommand(PGconn *conn, const char *query, bool echo)
* finish using them, pg_free(*table). *columns is a pointer into "spec",
* possibly to its NUL terminator.
*/
-static void
-split_table_columns_spec(const char *spec, int encoding,
- char **table, const char **columns)
+void
+splitTableColumnsSpec(const char *spec, int encoding,
+ char **table, const char **columns)
{
bool inquotes = false;
const char *cp = spec;
@@ -318,7 +318,7 @@ appendQualifiedRelation(PQExpBuffer buf, const char *spec,
return;
}
- split_table_columns_spec(spec, PQclientEncoding(conn), &table, &columns);
+ splitTableColumnsSpec(spec, PQclientEncoding(conn), &table, &columns);
/*
* Query must remain ABSOLUTELY devoid of unqualified names. This would
diff --git a/src/bin/scripts/common.h b/src/bin/scripts/common.h
index 7c3888cefd..b59ade0c39 100644
--- a/src/bin/scripts/common.h
+++ b/src/bin/scripts/common.h
@@ -48,6 +48,9 @@ extern void executeCommand(PGconn *conn, const char *query,
extern bool executeMaintenanceCommand(PGconn *conn, const char *query,
bool echo);
+extern void splitTableColumnsSpec(const char *spec, int encoding,
+ char **table, const char **columns);
+
extern void appendQualifiedRelation(PQExpBuffer buf, const char *name,
PGconn *conn, const char *progname, bool echo);
--
2.16.5
v5-0003-Always-use-a-catalog-query-to-discover-tables-to-.patchapplication/octet-stream; name=v5-0003-Always-use-a-catalog-query-to-discover-tables-to-.patchDownload
From caa66ac5de9a6de1c4c76f6450a4a0ee2f34e47c Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 23 Jan 2019 23:27:58 +0000
Subject: [PATCH v5 3/4] Always use a catalog query to discover tables to
process in vacuumdb.
---
src/bin/scripts/vacuumdb.c | 227 +++++++++++++++++++++++++++++++++------------
1 file changed, 170 insertions(+), 57 deletions(-)
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index ec7d0a326a..b8bf2e3ff5 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -19,6 +19,7 @@
#include "catalog/pg_class_d.h"
#include "common.h"
+#include "fe_utils/connect.h"
#include "fe_utils/simple_list.h"
#include "fe_utils/string_utils.h"
@@ -62,9 +63,7 @@ static void vacuum_all_databases(vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet);
static void prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo);
+ vacuumingOptions *vacopts, const char *table);
static void run_vacuum_command(PGconn *conn, const char *sql, bool echo,
const char *table, const char *progname, bool async);
@@ -359,13 +358,21 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet)
{
PQExpBufferData sql;
+ PQExpBufferData buf;
+ PQExpBufferData catalog_query;
+ PQExpBufferData cte;
+ PQExpBufferData table_clause;
+ PGresult *res;
PGconn *conn;
SimpleStringListCell *cell;
ParallelSlot *slots;
SimpleStringList dbtables = {NULL, NULL};
int i;
+ int ntups;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool tables_listed = false;
+ bool columns_listed = false;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -410,54 +417,172 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
+ /*
+ * Prepare the list of tables to process by querying the catalogs.
+ *
+ * Since we execute the constructed query with the default search_path
+ * (which could be unsafe), everything in this query must be fully
+ * qualified.
+ */
+ initPQExpBuffer(&cte);
+ initPQExpBuffer(&table_clause);
/*
- * If a table list is not provided and we're using multiple connections,
- * prepare the list of tables by querying the catalogs.
+ * First, build the WITH clause and the list of tables for the catalog
+ * query. The WITH clause is used to match any provided column lists with
+ * the generated qualified identifiers. The table list is used to filter
+ * for the tables provided via --table and to check that the table exists.
+ * If a listed table does not exist, the catalog query will fail, as will
+ * vacuumdb.
*/
- if (parallel && (!tables || !tables->head))
+ cell = tables ? tables->head : NULL;
+ while (cell != NULL)
{
- PQExpBufferData buf;
- PGresult *res;
- int ntups;
-
- initPQExpBuffer(&buf);
-
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
-
- ntups = PQntuples(res);
- for (i = 0; i < ntups; i++)
- {
- appendPQExpBufferStr(&buf,
- fmtQualifiedId(PQgetvalue(res, i, 1),
- PQgetvalue(res, i, 0)));
+ char *just_table;
+ const char *just_columns;
+
+ splitTableColumnsSpec(cell->val, PQclientEncoding(conn),
+ &just_table, &just_columns);
- simple_string_list_append(&dbtables, buf.data);
- resetPQExpBuffer(&buf);
+ /*
+ * First, handle the table list. The "regclass" identifier is used to
+ * look up the table's schema and name (using the default search_path)
+ * in order to generate a qualified identifier for the VACUUM or
+ * ANALYZE command.
+ */
+ if (!tables_listed)
+ {
+ appendPQExpBuffer(&table_clause, " AND (\n c.oid OPERATOR(pg_catalog.=) ");
+ tables_listed = true;
}
+ else
+ appendPQExpBuffer(&table_clause, " OR c.oid OPERATOR(pg_catalog.=) ");
- termPQExpBuffer(&buf);
- tables = &dbtables;
+ appendStringLiteralConn(&table_clause, just_table, conn);
+ appendPQExpBuffer(&table_clause, "::pg_catalog.regclass\n");
/*
- * If there are more connections than vacuumable relations, we don't
- * need to use them all.
+ * Now handle the column list, if provided. This generates a WITH
+ * clause that is used to map the table name and schema with the
+ * optional column list provided.
*/
+ if (just_columns != NULL && just_columns[0] != '\0')
+ {
+ if (!columns_listed)
+ {
+ appendPQExpBuffer(&cte, "WITH column_lists (table_name, column_list)"
+ " AS (\n VALUES (");
+ columns_listed = true;
+ }
+ else
+ appendPQExpBuffer(&cte, ",\n (");
+
+ appendStringLiteralConn(&cte, just_table, conn);
+ appendPQExpBuffer(&cte, ", ");
+ appendStringLiteralConn(&cte, just_columns, conn);
+ appendPQExpBuffer(&cte, ")");
+ }
+
+ pg_free(just_table);
+
+ cell = cell->next;
+ if (cell == NULL)
+ {
+ appendPQExpBuffer(&table_clause, " )\n");
+
+ if (columns_listed)
+ appendPQExpBuffer(&cte, "\n)\n");
+ }
+ }
+
+ /*
+ * Build the full catalog query. If the user specified tables with column
+ * lists, this consists of a WITH clause and a SELECT clause that filters
+ * based upon the table names. If the user specified tables but no column
+ * lists, just the SELECT clause with the table name filters is generated.
+ * If no tables or column lists were given, the query will exclude the
+ * table name filters as well.
+ */
+ initPQExpBuffer(&catalog_query);
+ if (columns_listed)
+ appendPQExpBufferStr(&catalog_query, cte.data);
+ termPQExpBuffer(&cte);
+
+ appendPQExpBuffer(&catalog_query, "SELECT c.relname, ns.nspname");
+
+ if (columns_listed)
+ appendPQExpBuffer(&catalog_query, ", column_lists.column_list");
+
+ appendPQExpBuffer(&catalog_query,
+ " FROM pg_catalog.pg_class c\n"
+ " JOIN pg_catalog.pg_namespace ns"
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+
+ if (columns_listed)
+ appendPQExpBuffer(&catalog_query, " LEFT JOIN column_lists"
+ " ON column_lists.table_name::pg_catalog.regclass OPERATOR(pg_catalog.=) c.oid\n");
+
+ appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) "])\n");
+
+ if (tables_listed)
+ appendPQExpBufferStr(&catalog_query, table_clause.data);
+ termPQExpBuffer(&table_clause);
+
+ /*
+ * Execute the catalog query. We use the default search_path for this
+ * query for consistency with table lookups done elsewhere by the user.
+ */
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
+
+ /*
+ * If no rows are returned, we are already done.
+ */
+ ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ /*
+ * Build qualified identifiers for each table, including the column list
+ * if given.
+ */
+ initPQExpBuffer(&buf);
+ for (i = 0; i < ntups; i++)
+ {
+ appendPQExpBufferStr(&buf,
+ fmtQualifiedId(PQgetvalue(res, i, 1),
+ PQgetvalue(res, i, 0)));
+
+ if (columns_listed && PQgetisnull(res, i, 2) == 0)
+ appendPQExpBufferStr(&buf, PQgetvalue(res, i, 2));
+
+ simple_string_list_append(&dbtables, buf.data);
+ resetPQExpBuffer(&buf);
+ }
+ termPQExpBuffer(&buf);
+
+ /*
+ * If there are more connections than vacuumable relations, we don't need
+ * to use them all.
+ */
+ if (parallel)
+ {
if (concurrentCons > ntups)
concurrentCons = ntups;
if (concurrentCons <= 1)
parallel = false;
- PQclear(res);
}
+ PQclear(res);
/*
* Setup the database connections. We reuse the connection we already have
@@ -493,10 +618,12 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
- cell = tables ? tables->head : NULL;
+ initPQExpBuffer(&sql);
+
+ cell = dbtables.head;
do
{
- const char *tabname = cell ? cell->val : NULL;
+ const char *tabname = cell->val;
ParallelSlot *free_slot;
if (CancelRequested)
@@ -529,12 +656,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
else
free_slot = slots;
- /*
- * Prepare the vacuum command. Note that in some cases this requires
- * query execution, so be sure to use the free connection.
- */
- prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname,
- tables == &dbtables, progname, echo);
+ prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname);
/*
* Execute the vacuum. If not in parallel mode, this terminates the
@@ -544,8 +666,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
run_vacuum_command(free_slot->connection, sql.data,
echo, tabname, progname, parallel);
- if (cell)
- cell = cell->next;
+ cell = cell->next;
} while (cell != NULL);
if (parallel)
@@ -658,9 +779,7 @@ vacuum_all_databases(vacuumingOptions *vacopts,
*/
static void
prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo)
+ vacuumingOptions *vacopts, const char *table)
{
const char *paren = " (";
const char *comma = ", ";
@@ -753,14 +872,8 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
}
}
- if (table)
- {
- appendPQExpBufferChar(sql, ' ');
- if (table_pre_qualified)
- appendPQExpBufferStr(sql, table);
- else
- appendQualifiedRelation(sql, table, conn, progname, echo);
- }
+ appendPQExpBufferChar(sql, ' ');
+ appendPQExpBufferStr(sql, table);
appendPQExpBufferChar(sql, ';');
}
--
2.16.5
v5-0004-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v5-0004-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From 9c1db3c811a0c1c2ba7cd2857bc5a1606a6fb014 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 22 Jan 2019 23:11:33 +0000
Subject: [PATCH v5 4/4] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 37 +++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 10 +++++++-
src/bin/scripts/vacuumdb.c | 54 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 99 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..29ccba2a5a 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,43 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ff0d97971b..b19a4abf26 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 35;
+use Test::More tests => 39;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -88,3 +88,11 @@ $node->issues_sql_like(
[ 'vacuumdb', '--analyze-only', '--table', 'vactable(b)', 'postgres' ],
qr/statement: ANALYZE public.vactable\(b\);/,
'vacuumdb --analyze-only with partial column list');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=2147483000', 'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=2147483001', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 2147483001/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index b8bf2e3ff5..7ca3766eef 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -406,6 +428,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -516,7 +552,9 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query,
" FROM pg_catalog.pg_class c\n"
" JOIN pg_catalog.pg_namespace ns"
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
if (columns_listed)
appendPQExpBuffer(&catalog_query, " LEFT JOIN column_lists"
@@ -530,6 +568,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBufferStr(&catalog_query, table_clause.data);
termPQExpBuffer(&table_clause);
+ if (vacopts->min_xid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_xid_age);
+
+ if (vacopts->min_mxid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_mxid_age);
+
/*
* Execute the catalog query. We use the default search_path for this
* query for consistency with table lookups done elsewhere by the user.
@@ -1188,6 +1238,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Thu, Jan 24, 2019 at 12:49:28AM +0000, Bossart, Nathan wrote:
Oh, wow. Thanks for pointing this out. I should have caught this.
With 0002, we are basically just throwing out the column lists
entirely as we obtain the qualified identifiers from the catalog
query. To fix this, I've added an optional CTE for tracking any
provided column lists. v5-0001 is your test patch for this case, and
v5-0002 splits out the work for split_table_columns_spec().
Committed the test portion for now, still reviewing the rest..
--
Michael
On Thu, Jan 24, 2019 at 12:49:28AM +0000, Bossart, Nathan wrote:
Oh, wow. Thanks for pointing this out. I should have caught this.
With 0002, we are basically just throwing out the column lists
entirely as we obtain the qualified identifiers from the catalog
query. To fix this, I've added an optional CTE for tracking any
provided column lists. v5-0001 is your test patch for this case, and
v5-0002 splits out the work for split_table_columns_spec().
I think that the query generation could be simplified by always using
the CTE if column lists are present or not, by associating NULL if no
column list is present, and by moving the regclass casting directly
into the CTE.
This way, for the following vacuumdb command with a set of tables
wanted:
vacuumdb --table aa --table 'bb(b)' --table 'cc(c)'
Then the query generated looks like that:
WITH column_lists (table_name, column_list) AS (
VALUES ('aa'::pg_catalog.regclass, NULL),
('bb'::pg_catalog.regclass, '(b)'),
('cc'::pg_catalog.regclass, '(c)')
)
SELECT c.relname, ns.nspname, column_lists.column_list
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace ns ON c.relnamespace
OPERATOR(pg_catalog.=) ns.oid
JOIN column_lists ON
column_lists.table_name
OPERATOR(pg_catalog.=) c.oid
WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array['r', 'm'])
ORDER BY c.relpages DESC;
So only the following parts are added:
- The CTE with a table and its column list.
- A join on pg_class.oid and column_lists.table_name.
The latest version of the patch is doing a double amount of work by
using a left join and an extra set of clauses in WHERE to fetch the
matching column list from the table name entry.
If no tables are listed, then we just finish with that:
SELECT c.relname, ns.nspname FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace ns ON c.relnamespace
OPERATOR(pg_catalog.=) ns.oid
WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array['r', 'm'])
ORDER BY c.relpages DESC;
--
Michael
On 1/27/19, 9:57 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
I think that the query generation could be simplified by always using
the CTE if column lists are present or not, by associating NULL if no
column list is present, and by moving the regclass casting directly
into the CTE.
Yes, this simplifies the code quite a bit. I did it this way in v6.
Nathan
Attachments:
v6-0001-Export-split_table_columns_spec-for-use-in-vacuum.patchapplication/octet-stream; name=v6-0001-Export-split_table_columns_spec-for-use-in-vacuum.patchDownload
From 3f6a3541618db15f2751561140b02b42eb9f2c77 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Mon, 28 Jan 2019 19:02:23 +0000
Subject: [PATCH v6 1/3] Export split_table_columns_spec() for use in vacuumdb.
---
src/bin/scripts/common.c | 8 ++++----
src/bin/scripts/common.h | 3 +++
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/src/bin/scripts/common.c b/src/bin/scripts/common.c
index 4215bc3d6e..7139b7c667 100644
--- a/src/bin/scripts/common.c
+++ b/src/bin/scripts/common.c
@@ -265,9 +265,9 @@ executeMaintenanceCommand(PGconn *conn, const char *query, bool echo)
* finish using them, pg_free(*table). *columns is a pointer into "spec",
* possibly to its NUL terminator.
*/
-static void
-split_table_columns_spec(const char *spec, int encoding,
- char **table, const char **columns)
+void
+splitTableColumnsSpec(const char *spec, int encoding,
+ char **table, const char **columns)
{
bool inquotes = false;
const char *cp = spec;
@@ -318,7 +318,7 @@ appendQualifiedRelation(PQExpBuffer buf, const char *spec,
return;
}
- split_table_columns_spec(spec, PQclientEncoding(conn), &table, &columns);
+ splitTableColumnsSpec(spec, PQclientEncoding(conn), &table, &columns);
/*
* Query must remain ABSOLUTELY devoid of unqualified names. This would
diff --git a/src/bin/scripts/common.h b/src/bin/scripts/common.h
index 7c3888cefd..b59ade0c39 100644
--- a/src/bin/scripts/common.h
+++ b/src/bin/scripts/common.h
@@ -48,6 +48,9 @@ extern void executeCommand(PGconn *conn, const char *query,
extern bool executeMaintenanceCommand(PGconn *conn, const char *query,
bool echo);
+extern void splitTableColumnsSpec(const char *spec, int encoding,
+ char **table, const char **columns);
+
extern void appendQualifiedRelation(PQExpBuffer buf, const char *name,
PGconn *conn, const char *progname, bool echo);
--
2.16.5
v6-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchapplication/octet-stream; name=v6-0002-Always-use-a-catalog-query-to-discover-tables-to-.patchDownload
From 4ee82e4bb492b82860a1e3852a41661fc0e7fc7d Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Mon, 28 Jan 2019 19:30:53 +0000
Subject: [PATCH v6 2/3] Always use a catalog query to discover tables to
process in vacuumdb.
---
src/bin/scripts/vacuumdb.c | 184 ++++++++++++++++++++++++++++++---------------
1 file changed, 125 insertions(+), 59 deletions(-)
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index ec7d0a326a..dccea23a36 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -19,6 +19,7 @@
#include "catalog/pg_class_d.h"
#include "common.h"
+#include "fe_utils/connect.h"
#include "fe_utils/simple_list.h"
#include "fe_utils/string_utils.h"
@@ -62,9 +63,7 @@ static void vacuum_all_databases(vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet);
static void prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo);
+ vacuumingOptions *vacopts, const char *table);
static void run_vacuum_command(PGconn *conn, const char *sql, bool echo,
const char *table, const char *progname, bool async);
@@ -359,13 +358,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
const char *progname, bool echo, bool quiet)
{
PQExpBufferData sql;
+ PQExpBufferData buf;
+ PQExpBufferData catalog_query;
+ PGresult *res;
PGconn *conn;
SimpleStringListCell *cell;
ParallelSlot *slots;
SimpleStringList dbtables = {NULL, NULL};
int i;
+ int ntups;
bool failed = false;
bool parallel = concurrentCons > 1;
+ bool tables_listed = false;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -410,54 +414,128 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
fflush(stdout);
}
- initPQExpBuffer(&sql);
+ /*
+ * Prepare the list of tables to process by querying the catalogs.
+ *
+ * Since we execute the constructed query with the default search_path
+ * (which could be unsafe), everything in this query must be fully
+ * qualified.
+ */
/*
- * If a table list is not provided and we're using multiple connections,
- * prepare the list of tables by querying the catalogs.
+ * First, build the WITH clause for the catalog query if any tables were
+ * specified. This is used to match any provided column lists with the
+ * generated qualified identifiers and to filter for the tables provided
+ * via --table. If a listed table does not exist, the catalog query will
+ * fail, as will vacuumdb.
*/
- if (parallel && (!tables || !tables->head))
+ initPQExpBuffer(&catalog_query);
+ cell = tables ? tables->head : NULL;
+ while (cell != NULL)
{
- PQExpBufferData buf;
- PGresult *res;
- int ntups;
-
- initPQExpBuffer(&buf);
-
- res = executeQuery(conn,
- "SELECT c.relname, ns.nspname"
- " FROM pg_class c, pg_namespace ns\n"
- " WHERE relkind IN ("
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) ")"
- " AND c.relnamespace = ns.oid\n"
- " ORDER BY c.relpages DESC;",
- progname, echo);
-
- ntups = PQntuples(res);
- for (i = 0; i < ntups; i++)
- {
- appendPQExpBufferStr(&buf,
- fmtQualifiedId(PQgetvalue(res, i, 1),
- PQgetvalue(res, i, 0)));
+ char *just_table;
+ const char *just_columns;
+
+ splitTableColumnsSpec(cell->val, PQclientEncoding(conn),
+ &just_table, &just_columns);
- simple_string_list_append(&dbtables, buf.data);
- resetPQExpBuffer(&buf);
+ if (!tables_listed)
+ {
+ appendPQExpBuffer(&catalog_query, "WITH listed_tables (table_oid, column_list)"
+ " AS (\n VALUES (");
+ tables_listed = true;
}
+ else
+ appendPQExpBuffer(&catalog_query, ",\n (");
- termPQExpBuffer(&buf);
- tables = &dbtables;
+ appendStringLiteralConn(&catalog_query, just_table, conn);
+ appendPQExpBuffer(&catalog_query, "::pg_catalog.regclass, ");
- /*
- * If there are more connections than vacuumable relations, we don't
- * need to use them all.
- */
+ if (just_columns && just_columns[0] != '\0')
+ appendStringLiteralConn(&catalog_query, just_columns, conn);
+ else
+ appendPQExpBufferStr(&catalog_query, "NULL");
+
+ appendPQExpBufferStr(&catalog_query, "::pg_catalog.text)");
+
+ pg_free(just_table);
+
+ cell = cell->next;
+ if (cell == NULL)
+ appendPQExpBuffer(&catalog_query, "\n)\n");
+ }
+
+ appendPQExpBuffer(&catalog_query, "SELECT c.relname, ns.nspname");
+
+ if (tables_listed)
+ appendPQExpBuffer(&catalog_query, ", listed_tables.column_list");
+
+ appendPQExpBuffer(&catalog_query,
+ " FROM pg_catalog.pg_class c\n"
+ " JOIN pg_catalog.pg_namespace ns"
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+
+ if (tables_listed)
+ appendPQExpBuffer(&catalog_query, " JOIN listed_tables"
+ " ON listed_tables.table_oid OPERATOR(pg_catalog.=) c.oid\n");
+
+ appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) "])\n");
+
+ /*
+ * Execute the catalog query. We use the default search_path for this
+ * query for consistency with table lookups done elsewhere by the user.
+ */
+ appendPQExpBuffer(&catalog_query, " ORDER BY c.relpages DESC;");
+ executeCommand(conn, "RESET search_path;", progname, echo);
+ res = executeQuery(conn, catalog_query.data, progname, echo);
+ termPQExpBuffer(&catalog_query);
+ PQclear(executeQuery(conn, ALWAYS_SECURE_SEARCH_PATH_SQL,
+ progname, echo));
+
+ /*
+ * If no rows are returned, we are already done.
+ */
+ ntups = PQntuples(res);
+ if (ntups == 0)
+ {
+ PQclear(res);
+ PQfinish(conn);
+ return;
+ }
+
+ /*
+ * Build qualified identifiers for each table, including the column list
+ * if given.
+ */
+ initPQExpBuffer(&buf);
+ for (i = 0; i < ntups; i++)
+ {
+ appendPQExpBufferStr(&buf,
+ fmtQualifiedId(PQgetvalue(res, i, 1),
+ PQgetvalue(res, i, 0)));
+
+ if (tables_listed && PQgetisnull(res, i, 2) == 0)
+ appendPQExpBufferStr(&buf, PQgetvalue(res, i, 2));
+
+ simple_string_list_append(&dbtables, buf.data);
+ resetPQExpBuffer(&buf);
+ }
+ termPQExpBuffer(&buf);
+
+ /*
+ * If there are more connections than vacuumable relations, we don't need
+ * to use them all.
+ */
+ if (parallel)
+ {
if (concurrentCons > ntups)
concurrentCons = ntups;
if (concurrentCons <= 1)
parallel = false;
- PQclear(res);
}
+ PQclear(res);
/*
* Setup the database connections. We reuse the connection we already have
@@ -493,10 +571,12 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
stage_commands[stage], progname, echo);
}
- cell = tables ? tables->head : NULL;
+ initPQExpBuffer(&sql);
+
+ cell = dbtables.head;
do
{
- const char *tabname = cell ? cell->val : NULL;
+ const char *tabname = cell->val;
ParallelSlot *free_slot;
if (CancelRequested)
@@ -529,12 +609,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
else
free_slot = slots;
- /*
- * Prepare the vacuum command. Note that in some cases this requires
- * query execution, so be sure to use the free connection.
- */
- prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname,
- tables == &dbtables, progname, echo);
+ prepare_vacuum_command(&sql, free_slot->connection, vacopts, tabname);
/*
* Execute the vacuum. If not in parallel mode, this terminates the
@@ -544,8 +619,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
run_vacuum_command(free_slot->connection, sql.data,
echo, tabname, progname, parallel);
- if (cell)
- cell = cell->next;
+ cell = cell->next;
} while (cell != NULL);
if (parallel)
@@ -658,9 +732,7 @@ vacuum_all_databases(vacuumingOptions *vacopts,
*/
static void
prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
- vacuumingOptions *vacopts, const char *table,
- bool table_pre_qualified,
- const char *progname, bool echo)
+ vacuumingOptions *vacopts, const char *table)
{
const char *paren = " (";
const char *comma = ", ";
@@ -753,14 +825,8 @@ prepare_vacuum_command(PQExpBuffer sql, PGconn *conn,
}
}
- if (table)
- {
- appendPQExpBufferChar(sql, ' ');
- if (table_pre_qualified)
- appendPQExpBufferStr(sql, table);
- else
- appendQualifiedRelation(sql, table, conn, progname, echo);
- }
+ appendPQExpBufferChar(sql, ' ');
+ appendPQExpBufferStr(sql, table);
appendPQExpBufferChar(sql, ';');
}
--
2.16.5
v6-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v6-0003-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From c6292a9fed7e44b7bb730f6127f6209646cc2e46 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Mon, 28 Jan 2019 19:33:24 +0000
Subject: [PATCH v6 3/3] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 37 +++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 10 +++++++-
src/bin/scripts/vacuumdb.c | 54 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 99 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..29ccba2a5a 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,43 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ff0d97971b..b19a4abf26 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 35;
+use Test::More tests => 39;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -88,3 +88,11 @@ $node->issues_sql_like(
[ 'vacuumdb', '--analyze-only', '--table', 'vactable(b)', 'postgres' ],
qr/statement: ANALYZE public.vactable\(b\);/,
'vacuumdb --analyze-only with partial column list');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=2147483000', 'postgres'],
+ qr/AND GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=2147483001', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 2147483001/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index dccea23a36..54c8551a22 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -403,6 +425,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -473,7 +509,9 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query,
" FROM pg_catalog.pg_class c\n"
" JOIN pg_catalog.pg_namespace ns"
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
if (tables_listed)
appendPQExpBuffer(&catalog_query, " JOIN listed_tables"
@@ -483,6 +521,18 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) "])\n");
+ if (vacopts->min_xid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_xid_age);
+
+ if (vacopts->min_mxid_age != 0)
+ appendPQExpBuffer(&catalog_query,
+ " AND GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n",
+ vacopts->min_mxid_age);
+
/*
* Execute the catalog query. We use the default search_path for this
* query for consistency with table lookups done elsewhere by the user.
@@ -1141,6 +1191,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Mon, Jan 28, 2019 at 09:58:17PM +0000, Bossart, Nathan wrote:
Yes, this simplifies the code quite a bit. I did it this way in
v6.
Thanks for the quick update. Things could have been made a bit more
simple by just using a for loop instead of while, even if the table
list can be NULL for database-wide operations (perhaps that's a matter
of taste..). prepare_vacuum_command() could be simplified further by
using the server version instead of the full connection string. The
comments at the top of the routine were not properly updated either,
and the last portion appending the relation name just needs one
appendPQExpBuffer() call instead of three separate calls. PQclear()
could have been moved closer to where all the query results have been
consumed, to make the whole more consistent.
Anyway, patches 1 and 2 have been merged, and committed after some
cleanup and adjustments. Patch 3 gets much easier now.
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
Why do need this part?
--
Michael
On 1/28/19, 6:35 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
Anyway, patches 1 and 2 have been merged, and committed after some
cleanup and adjustments. Patch 3 gets much easier now.
Thanks!
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"); + " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n" + " LEFT JOIN pg_catalog.pg_class t" + " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"); Why do need this part?
This is modeled after the query provided in the docs for preventing
transaction ID wraparound [0]https://www.postgresql.org/docs/devel/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND. I think the idea is to combine the
relation with its TOAST table so that it does not need to be
considered separately. The VACUUM commands generated in vacuumdb will
also process the corresponding TOAST table for the relation, anyway.
I noticed a behavior change from the catalog query patch that we
probably ought to fix. The "WHERE c.relkind IN ('r', 'm')" clause
seems sufficient to collect all vacuumable relations (TOAST tables are
handled when vacuuming the main relation, and partitioned tables are
handled by vacuuming the partitions individually), but it is not
sufficient to match the previous behavior when --table is used.
Previously, we did not filter by relkind at all when --table is used.
Instead, we let the server emit a WARNING when a relation that
couldn't be processed was specified.
Previous behavior:
~% vacuumdb -d postgres -t foreign_table
vacuumdb: vacuuming database "postgres"
WARNING: skipping "foreign_table" --- cannot vacuum non-tables or special system tables
~% vacuumdb -d postgres -t pg_toast.pg_toast_2600 --analyze-only
vacuumdb: vacuuming database "postgres"
WARNING: skipping "pg_toast_2600" --- cannot analyze non-tables or special system tables
Current behavior:
~% vacuumdb -d postgres -t foreign_table
vacuumdb: vacuuming database "postgres"
~% vacuumdb -d postgres -t pg_toast.pg_toast_2600 --analyze-only
vacuumdb: vacuuming database "postgres"
I think the simplest way to fix this is to remove the relkind clause
altogether when --table is used and to let the server decide whether
it should be processed. This effectively reinstates the previous
behavior so that users can specify TOAST tables, partitioned tables,
etc.
Unfortunately, this complicates the --min-xid-age and --min-mxid-age
patch a bit, as some of the relation types that can be vacuumed and/or
analyzed do not really have a transaction ID age. AFAICT the simplest
way to handle this case is to filter out relations with a relfrozenxid
or relminmxid of 0.
The v7 patch set implements these proposed approaches.
Nathan
[0]: https://www.postgresql.org/docs/devel/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND
Attachments:
v7-0001-Do-not-filter-by-relkind-in-vacuumdb-s-catalog-qu.patchapplication/octet-stream; name=v7-0001-Do-not-filter-by-relkind-in-vacuumdb-s-catalog-qu.patchDownload
From 2a119c39e642906fb3c69e8ce7932e3e0663c941 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 29 Jan 2019 20:03:09 +0000
Subject: [PATCH v7 1/2] Do not filter by relkind in vacuumdb's catalog query
if --table is used.
---
src/bin/scripts/t/100_vacuumdb.pl | 11 ++++++++++-
src/bin/scripts/vacuumdb.c | 13 ++++++++++---
2 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index ff0d97971b..7fd8881e13 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 35;
+use Test::More tests => 37;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -60,6 +60,15 @@ $node->command_ok(
[qw(vacuumdb -Zt pg_am(amname);ABORT postgres)],
'trailing command in "-t", with COLUMNS');
+$node->safe_psql(
+ 'postgres', q|
+ CREATE TABLE parted (a INT) PARTITION BY LIST (a);
+|);
+$node->issues_sql_like(
+ [ 'vacuumdb', '--analyze', 'postgres', '--table', 'parted'],
+ qr/statement: VACUUM \(ANALYZE\) public.parted;/,
+ 'vacuumdb --analyze processes partitioned table');
+
$node->safe_psql(
'postgres', q|
CREATE TABLE "need""q(uot" (")x" text);
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 05321edb8d..52853505b6 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -484,9 +484,16 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query, " JOIN listed_tables"
" ON listed_tables.table_oid OPERATOR(pg_catalog.=) c.oid\n");
- appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
- CppAsString2(RELKIND_RELATION) ", "
- CppAsString2(RELKIND_MATVIEW) "])\n");
+ /*
+ * If no tables were listed, filter for the relevant relation types. If
+ * tables were given via --table, don't bother filtering by relation type.
+ * Instead, let the server decide whether a given relation can be
+ * processed.
+ */
+ if (!tables_listed)
+ appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
+ CppAsString2(RELKIND_RELATION) ", "
+ CppAsString2(RELKIND_MATVIEW) "])\n");
/*
* Execute the catalog query. We use the default search_path for this
--
2.16.5
v7-0002-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v7-0002-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From e9eb8f613bdd3f41ea6fb178a2a91e588ba0cd89 Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Tue, 29 Jan 2019 19:55:29 +0000
Subject: [PATCH v7 2/2] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 37 ++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 10 +++++-
src/bin/scripts/vacuumdb.c | 66 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 111 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..29ccba2a5a 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,43 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 7fd8881e13..f8ba237dd9 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 37;
+use Test::More tests => 41;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -97,3 +97,11 @@ $node->issues_sql_like(
[ 'vacuumdb', '--analyze-only', '--table', 'vactable(b)', 'postgres' ],
qr/statement: ANALYZE public.vactable\(b\);/,
'vacuumdb --analyze-only with partial column list');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=2147483000', 'postgres'],
+ qr/WHERE GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=2147483001', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 2147483001/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 52853505b6..76e8a8319f 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -370,6 +392,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
bool failed = false;
bool parallel = concurrentCons > 1;
bool tables_listed = false;
+ bool has_where = false;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -403,6 +426,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -477,7 +514,9 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query,
" FROM pg_catalog.pg_class c\n"
" JOIN pg_catalog.pg_namespace ns"
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
/* Used to match the tables listed by the user */
if (tables_listed)
@@ -491,9 +530,32 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
* processed.
*/
if (!tables_listed)
+ {
appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) "])\n");
+ has_where = true;
+ }
+
+ if (vacopts->min_xid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n"
+ " AND c.relfrozenxid OPERATOR(pg_catalog.!=) 0\n",
+ has_where ? "AND" : "WHERE", vacopts->min_xid_age);
+ has_where = true;
+ }
+
+ if (vacopts->min_mxid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n"
+ " AND c.relminmxid OPERATOR(pg_catalog.!=) 0\n",
+ has_where ? "AND" : "WHERE", vacopts->min_mxid_age);
+ has_where = true;
+ }
/*
* Execute the catalog query. We use the default search_path for this
@@ -1152,6 +1214,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Tue, Jan 29, 2019 at 09:48:18PM +0000, Bossart, Nathan wrote:
On 1/28/19, 6:35 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"); + " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n" + " LEFT JOIN pg_catalog.pg_class t" + " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"); Why do need this part?This is modeled after the query provided in the docs for preventing
transaction ID wraparound [0]. I think the idea is to combine the
relation with its TOAST table so that it does not need to be
considered separately. The VACUUM commands generated in vacuumdb will
also process the corresponding TOAST table for the relation, anyway.
Oh, OK. This makes sense. It would be nice to add a comment in the
patch and to document this calculation method in the docs of
vacuumdb.
I noticed a behavior change from the catalog query patch that we
probably ought to fix. The "WHERE c.relkind IN ('r', 'm')" clause
seems sufficient to collect all vacuumable relations (TOAST tables are
handled when vacuuming the main relation, and partitioned tables are
handled by vacuuming the partitions individually), but it is not
sufficient to match the previous behavior when --table is used.
Previously, we did not filter by relkind at all when --table is used.
Instead, we let the server emit a WARNING when a relation that
couldn't be processed was specified.
Indeed, the WARNING can be useful for some users when trying to work
on an incorrect relation kind, especially when not using --verbose.
Fixed after adding a test with command_checks_all.
Unfortunately, this complicates the --min-xid-age and --min-mxid-age
patch a bit, as some of the relation types that can be vacuumed and/or
analyzed do not really have a transaction ID age. AFAICT the simplest
way to handle this case is to filter out relations with a relfrozenxid
or relminmxid of 0.
We should be able to live with that.
--
Michael
On 1/29/19, 4:47 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
On Tue, Jan 29, 2019 at 09:48:18PM +0000, Bossart, Nathan wrote:
On 1/28/19, 6:35 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"); + " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n" + " LEFT JOIN pg_catalog.pg_class t" + " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n"); Why do need this part?This is modeled after the query provided in the docs for preventing
transaction ID wraparound [0]. I think the idea is to combine the
relation with its TOAST table so that it does not need to be
considered separately. The VACUUM commands generated in vacuumdb will
also process the corresponding TOAST table for the relation, anyway.Oh, OK. This makes sense. It would be nice to add a comment in the
patch and to document this calculation method in the docs of
vacuumdb.
Sure, this is added in v8.
I noticed a behavior change from the catalog query patch that we
probably ought to fix. The "WHERE c.relkind IN ('r', 'm')" clause
seems sufficient to collect all vacuumable relations (TOAST tables are
handled when vacuuming the main relation, and partitioned tables are
handled by vacuuming the partitions individually), but it is not
sufficient to match the previous behavior when --table is used.
Previously, we did not filter by relkind at all when --table is used.
Instead, we let the server emit a WARNING when a relation that
couldn't be processed was specified.Indeed, the WARNING can be useful for some users when trying to work
on an incorrect relation kind, especially when not using --verbose.
Fixed after adding a test with command_checks_all.
Thanks. Something else I noticed is that we do not retrieve foreign
tables and partitioned tables for --analyze and --analyze-only.
However, that has long been the case for parallel mode, and this issue
should probably get its own thread.
Nathan
Attachments:
v8-0001-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchapplication/octet-stream; name=v8-0001-Add-min-xid-age-and-min-mxid-age-options-to-vacuu.patchDownload
From 547e4261d804fb82015dbb5c45ed3bfe54e9ca7f Mon Sep 17 00:00:00 2001
From: Nathan Bossart <bossartn@amazon.com>
Date: Wed, 30 Jan 2019 17:35:31 +0000
Subject: [PATCH v8 1/1] Add --min-xid-age and --min-mxid-age options to
vacuumdb.
---
doc/src/sgml/ref/vacuumdb.sgml | 53 +++++++++++++++++++++++++++
src/bin/scripts/t/100_vacuumdb.pl | 10 +++++-
src/bin/scripts/vacuumdb.c | 75 ++++++++++++++++++++++++++++++++++++++-
3 files changed, 136 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..87b3349a59 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,59 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see <xref
+ linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <para>
+ For the purposes of this option, the multixact ID age of a relation is
+ the greatest of the ages of the main relation and its associated
+ <acronym>TOAST</acronym> table, if one exists. Since the commands
+ issued by <application>vacuumdb</application> will also process the
+ <acronym>TOAST</acronym> table for the relation if necessary, it does
+ not need to be considered separately.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 9.5 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a transaction
+ ID age of at least <replaceable class="parameter">xid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ transaction ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <para>
+ For the purposes of this option, the transaction ID age of a relation is
+ the greatest of the ages of the main relation and its associated
+ <acronym>TOAST</acronym> table, if one exists. Since the commands
+ issued by <application>vacuumdb</application> will also process the
+ <acronym>TOAST</acronym> table for the relation if necessary, it does
+ not need to be considered separately.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running PostgreSQL 7.2 and
+ later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 5e87af2d51..a560dba53f 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 38;
+use Test::More tests => 42;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -95,3 +95,11 @@ $node->command_checks_all(
[qr/^.*vacuuming database "postgres"/],
[qr/^WARNING.*cannot vacuum non-tables or special system tables/s],
'vacuumdb with view');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table=pg_class', '--min-mxid-age=2147483000', 'postgres'],
+ qr/WHERE GREATEST\(pg_catalog.mxid_age\(c.relminmxid\), pg_catalog.mxid_age\(t.relminmxid\)\) OPERATOR\(pg_catalog.>=\) 2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age=2147483001', 'postgres' ],
+ qr/AND GREATEST\(pg_catalog.age\(c.relfrozenxid\), pg_catalog.age\(t.relfrozenxid\)\) OPERATOR\(pg_catalog.>=\) 2147483001/,
+ 'vacuumdb --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 40ba8283a2..dc1e4d3e90 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -370,6 +392,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
bool failed = false;
bool parallel = concurrentCons > 1;
bool tables_listed = false;
+ bool has_where = false;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -403,6 +426,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 70200)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 7.2\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90500)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.5\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -477,7 +514,9 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query,
" FROM pg_catalog.pg_class c\n"
" JOIN pg_catalog.pg_namespace ns"
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
/* Used to match the tables listed by the user */
if (tables_listed)
@@ -491,9 +530,41 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
* processed in which case the user will know about it.
*/
if (!tables_listed)
+ {
appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) "])\n");
+ has_where = true;
+ }
+
+ /*
+ * For --min-xid-age and --min-mxid-age, the age of the relation is the
+ * greatest of the ages of the main relation and its associated TOAST
+ * table. The commands generated by vacuumdb will also process the TOAST
+ * table for the relation if necessary, so it does not need to be
+ * considered separately. Note that ANALYZE is always skipped on TOAST
+ * tables because the toaster always uses hardcoded index access and
+ * statistics are totally unimportant for TOAST relations.
+ */
+ if (vacopts->min_xid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) OPERATOR(pg_catalog.>=) %d\n"
+ " AND c.relfrozenxid OPERATOR(pg_catalog.!=) 0\n",
+ has_where ? "AND" : "WHERE", vacopts->min_xid_age);
+ has_where = true;
+ }
+
+ if (vacopts->min_mxid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=) %d\n"
+ " AND c.relminmxid OPERATOR(pg_catalog.!=) 0\n",
+ has_where ? "AND" : "WHERE", vacopts->min_mxid_age);
+ has_where = true;
+ }
/*
* Execute the catalog query. We use the default search_path for this
@@ -1152,6 +1223,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
--
2.16.5
On Wed, Jan 30, 2019 at 05:45:58PM +0000, Bossart, Nathan wrote:
On 1/29/19, 4:47 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
Oh, OK. This makes sense. It would be nice to add a comment in the
patch and to document this calculation method in the docs of
vacuumdb.Sure, this is added in v8.
Thanks, Nathan.
Something which was not correct in the patch is the compatibility of
the query. xid <> xid has been added in 9.6, so the new options will
not be able to work with older versions. The versions marked as
compatible in the last patch came from the age-ing functions, but you
added direct comparisons with relfrozenxid and relminmxid in the
latest versions of the patch. This implementation goes down a couple
of released versions, which is useful enough in my opinion, so I would
keep it as-is.
I have added as well some markups around "PostgreSQL" in the docs, and
extra casts for the integer/xid values of the query. The test
patterns are also simplified, and I added tests for incorrect values
of --min-xid-age and --min-mxid-age. Does that look correct to you?
Thanks. Something else I noticed is that we do not retrieve foreign
tables and partitioned tables for --analyze and --analyze-only.
However, that has long been the case for parallel mode, and this issue
should probably get its own thread.
Good point, this goes down a couple of releases, and statistics on
both may be useful to compile for a system-wide operation. Spawning a
separate thread looks adapted to me.
--
Michael
Attachments:
vacuumdb-min-age-v9.patchtext/x-diff; charset=us-asciiDownload
diff --git a/doc/src/sgml/ref/vacuumdb.sgml b/doc/src/sgml/ref/vacuumdb.sgml
index f304627802..41c7f3df79 100644
--- a/doc/src/sgml/ref/vacuumdb.sgml
+++ b/doc/src/sgml/ref/vacuumdb.sgml
@@ -172,6 +172,60 @@ PostgreSQL documentation
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>--min-mxid-age <replaceable class="parameter">mxid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a multixact
+ ID age of at least <replaceable class="parameter">mxid_age</replaceable>.
+ This setting is useful for prioritizing tables to process to prevent
+ multixact ID wraparound (see
+ <xref linkend="vacuum-for-multixact-wraparound"/>).
+ </para>
+ <para>
+ For the purposes of this option, the multixact ID age of a relation is
+ the greatest of the ages of the main relation and its associated
+ <acronym>TOAST</acronym> table, if one exists. Since the commands
+ issued by <application>vacuumdb</application> will also process the
+ <acronym>TOAST</acronym> table for the relation if necessary, it does
+ not need to be considered separately.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running
+ <productname>PostgreSQL</productname> 9.6 and later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--min-xid-age <replaceable class="parameter">xid_age</replaceable></option></term>
+ <listitem>
+ <para>
+ Only execute the vacuum or analyze commands on tables with a
+ transaction ID age of at least
+ <replaceable class="parameter">xid_age</replaceable>. This setting
+ is useful for prioritizing tables to process to prevent transaction
+ ID wraparound (see <xref linkend="vacuum-for-wraparound"/>).
+ </para>
+ <para>
+ For the purposes of this option, the transaction ID age of a relation
+ is the greatest of the ages of the main relation and its associated
+ <acronym>TOAST</acronym> table, if one exists. Since the commands
+ issued by <application>vacuumdb</application> will also process the
+ <acronym>TOAST</acronym> table for the relation if necessary, it does
+ not need to be considered separately.
+ </para>
+ <note>
+ <para>
+ This option is only available for servers running
+ <productname>PostgreSQL</productname> 9.6 and later.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-q</option></term>
<term><option>--quiet</option></term>
diff --git a/src/bin/scripts/t/100_vacuumdb.pl b/src/bin/scripts/t/100_vacuumdb.pl
index 5e87af2d51..7f3a9b14a9 100644
--- a/src/bin/scripts/t/100_vacuumdb.pl
+++ b/src/bin/scripts/t/100_vacuumdb.pl
@@ -3,7 +3,7 @@ use warnings;
use PostgresNode;
use TestLib;
-use Test::More tests => 38;
+use Test::More tests => 44;
program_help_ok('vacuumdb');
program_version_ok('vacuumdb');
@@ -95,3 +95,20 @@ $node->command_checks_all(
[qr/^.*vacuuming database "postgres"/],
[qr/^WARNING.*cannot vacuum non-tables or special system tables/s],
'vacuumdb with view');
+$node->command_fails(
+ [ 'vacuumdb', '--table', 'vactable', '--min-mxid-age', '0',
+ 'postgres'],
+ 'vacuumdb --min-mxid-age with incorrect value');
+$node->command_fails(
+ [ 'vacuumdb', '--table', 'vactable', '--min-xid-age', '0',
+ 'postgres'],
+ 'vacuumdb --min-xid-age with incorrect value');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--table', 'vactable', '--min-mxid-age', '2147483000',
+ 'postgres'],
+ qr/GREATEST.*relminmxid.*2147483000/,
+ 'vacuumdb --table --min-mxid-age');
+$node->issues_sql_like(
+ [ 'vacuumdb', '--min-xid-age', '2147483001', 'postgres' ],
+ qr/GREATEST.*relfrozenxid.*2147483001/,
+ 'vacuumdb --table --min-xid-age');
diff --git a/src/bin/scripts/vacuumdb.c b/src/bin/scripts/vacuumdb.c
index 40ba8283a2..cb503569bb 100644
--- a/src/bin/scripts/vacuumdb.c
+++ b/src/bin/scripts/vacuumdb.c
@@ -43,6 +43,8 @@ typedef struct vacuumingOptions
bool freeze;
bool disable_page_skipping;
bool skip_locked;
+ int min_xid_age;
+ int min_mxid_age;
} vacuumingOptions;
@@ -113,6 +115,8 @@ main(int argc, char *argv[])
{"analyze-in-stages", no_argument, NULL, 3},
{"disable-page-skipping", no_argument, NULL, 4},
{"skip-locked", no_argument, NULL, 5},
+ {"min-xid-age", required_argument, NULL, 6},
+ {"min-mxid-age", required_argument, NULL, 7},
{NULL, 0, NULL, 0}
};
@@ -222,6 +226,24 @@ main(int argc, char *argv[])
case 5:
vacopts.skip_locked = true;
break;
+ case 6:
+ vacopts.min_xid_age = atoi(optarg);
+ if (vacopts.min_xid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum transaction ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
+ case 7:
+ vacopts.min_mxid_age = atoi(optarg);
+ if (vacopts.min_mxid_age <= 0)
+ {
+ fprintf(stderr, _("%s: minimum multixact ID age must be at least 1\n"),
+ progname);
+ exit(1);
+ }
+ break;
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -370,6 +392,7 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
bool failed = false;
bool parallel = concurrentCons > 1;
bool tables_listed = false;
+ bool has_where = false;
const char *stage_commands[] = {
"SET default_statistics_target=1; SET vacuum_cost_delay=0;",
"SET default_statistics_target=10; RESET vacuum_cost_delay;",
@@ -403,6 +426,20 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
exit(1);
}
+ if (vacopts->min_xid_age != 0 && PQserverVersion(conn) < 90600)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.6\n"),
+ progname, "--min-xid-age");
+ exit(1);
+ }
+
+ if (vacopts->min_mxid_age != 0 && PQserverVersion(conn) < 90600)
+ {
+ fprintf(stderr, _("%s: cannot use the \"%s\" option on server versions older than PostgreSQL 9.6\n"),
+ progname, "--min-mxid-age");
+ exit(1);
+ }
+
if (!quiet)
{
if (stage != ANALYZE_NO_STAGE)
@@ -477,7 +514,9 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
appendPQExpBuffer(&catalog_query,
" FROM pg_catalog.pg_class c\n"
" JOIN pg_catalog.pg_namespace ns"
- " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n");
+ " ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid\n"
+ " LEFT JOIN pg_catalog.pg_class t"
+ " ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid\n");
/* Used to match the tables listed by the user */
if (tables_listed)
@@ -491,9 +530,43 @@ vacuum_one_database(const char *dbname, vacuumingOptions *vacopts,
* processed in which case the user will know about it.
*/
if (!tables_listed)
+ {
appendPQExpBuffer(&catalog_query, " WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array["
CppAsString2(RELKIND_RELATION) ", "
CppAsString2(RELKIND_MATVIEW) "])\n");
+ has_where = true;
+ }
+
+ /*
+ * For --min-xid-age and --min-mxid-age, the age of the relation is the
+ * greatest of the ages of the main relation and its associated TOAST
+ * table. The commands generated by vacuumdb will also process the TOAST
+ * table for the relation if necessary, so it does not need to be
+ * considered separately.
+ */
+ if (vacopts->min_xid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.age(c.relfrozenxid),"
+ " pg_catalog.age(t.relfrozenxid)) "
+ " OPERATOR(pg_catalog.>=) '%d'::pg_catalog.int4\n"
+ " AND c.relfrozenxid OPERATOR(pg_catalog.!=)"
+ " '0'::pg_catalog.xid\n",
+ has_where ? "AND" : "WHERE", vacopts->min_xid_age);
+ has_where = true;
+ }
+
+ if (vacopts->min_mxid_age != 0)
+ {
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=)"
+ " '%d'::pg_catalog.int4\n"
+ " AND c.relfrozenxid OPERATOR(pg_catalog.!=)"
+ " '0'::pg_catalog.xid\n",
+ has_where ? "AND" : "WHERE", vacopts->min_mxid_age);
+ has_where = true;
+ }
/*
* Execute the catalog query. We use the default search_path for this
@@ -1152,6 +1225,8 @@ help(const char *progname)
printf(_(" -f, --full do full vacuuming\n"));
printf(_(" -F, --freeze freeze row transaction information\n"));
printf(_(" -j, --jobs=NUM use this many concurrent connections to vacuum\n"));
+ printf(_(" --min-mxid-age=MXID_AGE minimum multixact ID age of tables to vacuum\n"));
+ printf(_(" --min-xid-age=XID_AGE minimum transaction ID age of tables to vacuum\n"));
printf(_(" -q, --quiet don't write any messages\n"));
printf(_(" --skip-locked skip relations that cannot be immediately locked\n"));
printf(_(" -t, --table='TABLE[(COLUMNS)]' vacuum specific table(s) only\n"));
On 1/30/19, 6:04 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
Something which was not correct in the patch is the compatibility of
the query. xid <> xid has been added in 9.6, so the new options will
not be able to work with older versions. The versions marked as
compatible in the last patch came from the age-ing functions, but you
added direct comparisons with relfrozenxid and relminmxid in the
latest versions of the patch. This implementation goes down a couple
of released versions, which is useful enough in my opinion, so I would
keep it as-is.
Agreed. Thanks for catching this.
I have added as well some markups around "PostgreSQL" in the docs, and
extra casts for the integer/xid values of the query. The test
patterns are also simplified, and I added tests for incorrect values
of --min-xid-age and --min-mxid-age. Does that look correct to you?
It looks good to me. The only thing I noticed is the use of
relfrozenxid instead of relminmxid here:
+ appendPQExpBuffer(&catalog_query,
+ " %s GREATEST(pg_catalog.mxid_age(c.relminmxid),"
+ " pg_catalog.mxid_age(t.relminmxid)) OPERATOR(pg_catalog.>=)"
+ " '%d'::pg_catalog.int4\n"
+ " AND c.relfrozenxid OPERATOR(pg_catalog.!=)"
+ " '0'::pg_catalog.xid\n",
+ has_where ? "AND" : "WHERE", vacopts->min_mxid_age);
However, that may still work as intended.
Nathan
On Thu, Jan 31, 2019 at 02:28:05AM +0000, Bossart, Nathan wrote:
It looks good to me. The only thing I noticed is the use of
relfrozenxid instead of relminmxid here:
Fixed and committed. So we are finally done here, except for the
debate with the relation size.
--
Michael
On 1/30/19, 8:21 PM, "Michael Paquier" <michael@paquier.xyz> wrote:
Fixed and committed. So we are finally done here, except for the
debate with the relation size.
Thanks for the diligent reviews, as always. I don't plan to pick up
--min-relation-size right now, but I may attempt it again in a future
commitfest.
Nathan