Use of RangeVar for partitioned tables in autovacuum

Started by Michael Paquierover 8 years ago5 messages
#1Michael Paquier
michael.paquier@gmail.com
1 attachment(s)

Hi all,

I have been looking more closely at the problem in $subject, that I
have mentioned a couple of times, like here:
/messages/by-id/CAB7nPqQa37OUne_rJzpmWC4EXQaLX9f27-Ma_-rSFL_3mj+HFQ@mail.gmail.com

As of HEAD, the RangeVar defined in calls of vacuum() is used for
error reporting, only in two cases: for autoanalyze and autovacuum
when reporting to users that a lock cannot be taken on a relation. The
thing is that autovacuum has the good idea to call vacuum() on only
one relation at the same time, with always a relation OID and a
RangeVar defined, so the code currently on HEAD is basically fine with
its error reporting, because get_rel_oids() will always work on the
relation OID with its RangeVar, and because code paths with manual
VACUUMs don't use the RangeVar for any reports.

While v10 is safe, I think that this is wrong in the long-term and
that may be a cause of bugs if people begin to generate reports for
manual VACUUMs, which is plausible in my opinion. The patch attached,
is what one solution would look like if we would like to make things
more robust as long as manual VACUUM commands can only specify one
relation at the same time. I have found that tracking the parent OID
by tweaking a bit get_rel_oids() was the most elegant solution. Please
note that this range of problems is something that I think is better
solved with the infrastructure that support for VACUUM with multiple
relations includes (last version here
/messages/by-id/766556DD-AA3C-42F7-ACF4-5DC97F41F151@amazon.com).
So I don't think that the patch attached should be integrated, still I
am attaching it to satisfy the curiosity of anybody looking at this
message.

In conclusion, I think that the open item of $subject should be
removed from the list, and we should try to get the multi-VACUUM patch
in to cover any future problems. I'll do so if there are no
objections.

One comment on top of vacuum() is wrong by the way: in the case of a
manual VACUUM or ANALYZE, a NULL RangeVar is used if no relation is
specified in the command.

Thanks,
--
Michael

Attachments:

vac-rangevar-v1.patchapplication/octet-stream; name=vac-rangevar-v1.patchDownload
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index faa181207a..4031fae57d 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -67,7 +67,7 @@ static BufferAccessStrategy vac_strategy;
 
 
 /* non-export function prototypes */
-static List *get_rel_oids(Oid relid, const RangeVar *vacrel);
+static List *get_rel_oids(Oid *relid, const RangeVar *vacrel);
 static void vac_truncate_clog(TransactionId frozenXID,
 				  MultiXactId minMulti,
 				  TransactionId lastSaneFrozenXid,
@@ -129,8 +129,12 @@ ExecVacuum(VacuumStmt *vacstmt, bool isTopLevel)
  * options is a bitmask of VacuumOption flags, indicating what to do.
  *
  * relid, if not InvalidOid, indicate the relation to process; otherwise,
- * the RangeVar is used.  (The latter must always be passed, because it's
- * used for error messages.)
+ * the RangeVar is used.  The latter must be passed because it is used
+ * for error reporting, still it will not be used for relations vacuumed
+ * which have been dragged into the process due to inheritance, like
+ * child partitions.  Note that RangeVar is NULL if relid is InvalidOid,
+ * which happens in the case of manual VACUUM commands not specifying a
+ * list of relations.
  *
  * params contains a set of parameters that can be used to customize the
  * behavior.
@@ -229,7 +233,7 @@ vacuum(int options, RangeVar *relation, Oid relid, VacuumParams *params,
 	 * Build list of relations to process, unless caller gave us one. (If we
 	 * build one, we put it in vac_context for safekeeping.)
 	 */
-	relations = get_rel_oids(relid, relation);
+	relations = get_rel_oids(&relid, relation);
 
 	/*
 	 * Decide whether we need to start/commit our own transactions.
@@ -297,11 +301,48 @@ vacuum(int options, RangeVar *relation, Oid relid, VacuumParams *params,
 		 */
 		foreach(cur, relations)
 		{
-			Oid			relid = lfirst_oid(cur);
+			Oid			proc_relid = lfirst_oid(cur);
+			RangeVar   *rv;
+
+			/*
+			 * Determine which RangeVar to use for reporting should the
+			 * relation referenced by the OID listed here should be
+			 * missing.
+			 */
+			if (IsAutoVacuumWorkerProcess())
+			{
+				/*
+				 * Autovacuum workers work on a pre-relation basis, so there
+				 * is always a RangeVar to rely on
+				 */
+				Assert(relation != NULL);
+				rv = relation;
+			}
+			else if (relation)
+			{
+				/*
+				 * A manual command is being run with a relation specified.
+				 * Be careful that the relation being processed here may
+				 * be a partitioned table, in which case the RangeVar
+				 * specified by caller can only be used if the relation
+				 * processed here is the parent itself. Hence check if the
+				 * relation being processed is the original relation itself
+				 * and use its RangeVar if those are the same.
+				 */
+				rv = proc_relid == relid ? relation : NULL;
+			}
+			else
+			{
+				/*
+				 * This is the case of a manual command which does not specify
+				 * a relation to work on, so just use nothing.
+				 */
+				rv = NULL;
+			}
 
 			if (options & VACOPT_VACUUM)
 			{
-				if (!vacuum_rel(relid, relation, options, params))
+				if (!vacuum_rel(proc_relid, rv, options, params))
 					continue;
 			}
 
@@ -318,7 +359,7 @@ vacuum(int options, RangeVar *relation, Oid relid, VacuumParams *params,
 					PushActiveSnapshot(GetTransactionSnapshot());
 				}
 
-				analyze_rel(relid, relation, options, params,
+				analyze_rel(proc_relid, rv, options, params,
 							va_cols, in_outer_xact, vac_strategy);
 
 				if (use_own_xacts)
@@ -376,29 +417,36 @@ vacuum(int options, RangeVar *relation, Oid relid, VacuumParams *params,
  * Build a list of Oids for each relation to be processed
  *
  * The list is built in vac_context so that it will survive across our
- * per-relation transactions.
+ * per-relation transactions. If relid is defined as InvalidOid by the
+ * caller, then it gets filled with the OID of the parent relation
+ * used for the potential inheritance scanning for partitioned tables.
  */
 static List *
-get_rel_oids(Oid relid, const RangeVar *vacrel)
+get_rel_oids(Oid *relid, const RangeVar *vacrel)
 {
 	List	   *oid_list = NIL;
 	MemoryContext oldcontext;
 
 	/* OID supplied by VACUUM's caller? */
-	if (OidIsValid(relid))
+	if (OidIsValid(*relid))
 	{
+		/* normal course of events for an autovacuum worker */
+		Assert(IsAutoVacuumWorkerProcess());
 		oldcontext = MemoryContextSwitchTo(vac_context);
-		oid_list = lappend_oid(oid_list, relid);
+		oid_list = lappend_oid(oid_list, *relid);
 		MemoryContextSwitchTo(oldcontext);
 	}
 	else if (vacrel)
 	{
 		/* Process a specific relation */
-		Oid			relid;
+		Oid			parentoid;
 		HeapTuple	tuple;
 		Form_pg_class classForm;
 		bool		include_parts;
 
+		/* manual VACUUM running here */
+		Assert(!IsAutoVacuumWorkerProcess());
+
 		/*
 		 * Since we don't take a lock here, the relation might be gone, or the
 		 * RangeVar might no longer refer to the OID we look up here.  In the
@@ -408,15 +456,15 @@ get_rel_oids(Oid relid, const RangeVar *vacrel)
 		 * going to commit this transaction and begin a new one between now
 		 * and then.
 		 */
-		relid = RangeVarGetRelid(vacrel, NoLock, false);
+		parentoid = RangeVarGetRelid(vacrel, NoLock, false);
 
 		/*
 		 * To check whether the relation is a partitioned table, fetch its
 		 * syscache entry.
 		 */
-		tuple = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+		tuple = SearchSysCache1(RELOID, ObjectIdGetDatum(parentoid));
 		if (!HeapTupleIsValid(tuple))
-			elog(ERROR, "cache lookup failed for relation %u", relid);
+			elog(ERROR, "cache lookup failed for relation %u", parentoid);
 		classForm = (Form_pg_class) GETSTRUCT(tuple);
 		include_parts = (classForm->relkind == RELKIND_PARTITIONED_TABLE);
 		ReleaseSysCache(tuple);
@@ -430,10 +478,12 @@ get_rel_oids(Oid relid, const RangeVar *vacrel)
 		oldcontext = MemoryContextSwitchTo(vac_context);
 		if (include_parts)
 			oid_list = list_concat(oid_list,
-								   find_all_inheritors(relid, NoLock, NULL));
+								   find_all_inheritors(parentoid, NoLock, NULL));
 		else
-			oid_list = lappend_oid(oid_list, relid);
+			oid_list = lappend_oid(oid_list, parentoid);
 		MemoryContextSwitchTo(oldcontext);
+
+		*relid = parentoid;
 	}
 	else
 	{
@@ -445,6 +495,9 @@ get_rel_oids(Oid relid, const RangeVar *vacrel)
 		HeapScanDesc scan;
 		HeapTuple	tuple;
 
+		/* manual vacuum running here */
+		Assert(!IsAutoVacuumWorkerProcess());
+
 		pgclass = heap_open(RelationRelationId, AccessShareLock);
 
 		scan = heap_beginscan_catalog(pgclass, 0, NULL);
#2Bossart, Nathan
bossartn@amazon.com
In reply to: Michael Paquier (#1)
Re: Use of RangeVar for partitioned tables in autovacuum

On 9/26/17, 9:28 PM, "Michael Paquier" <michael.paquier@gmail.com> wrote:

In conclusion, I think that the open item of $subject should be
removed from the list, and we should try to get the multi-VACUUM patch
in to cover any future problems. I'll do so if there are no
objections.

If someone did want to add logging for vacuum_rel() and analyze_rel() in
v10 after your patch was applied, wouldn't the NULL RangeVars force us to
skip the new log statements for partitions? I think we might instead
want to back-patch the VacuumRelation infrastructure so that we can
appropriately log for partitions.

However, I'm dubious that it is necessary to make such a big change so
close to release for hypothetical log statements. So, in the end, I agree
with you.

Nathan

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Michael Paquier
michael.paquier@gmail.com
In reply to: Bossart, Nathan (#2)
Re: Use of RangeVar for partitioned tables in autovacuum

On Thu, Sep 28, 2017 at 3:11 AM, Bossart, Nathan <bossartn@amazon.com> wrote:

On 9/26/17, 9:28 PM, "Michael Paquier" <michael.paquier@gmail.com> wrote:

In conclusion, I think that the open item of $subject should be
removed from the list, and we should try to get the multi-VACUUM patch
in to cover any future problems. I'll do so if there are no
objections.

If someone did want to add logging for vacuum_rel() and analyze_rel() in
v10 after your patch was applied, wouldn't the NULL RangeVars force us to
skip the new log statements for partitions? I think we might instead
want to back-patch the VacuumRelation infrastructure so that we can
appropriately log for partitions.

Yes, in this case that would be a problem. But that problem does not
exist yet. If that was to happen, something like the patch I attached
would be actually enough. It is simple and non-intrusive as well.

However, I'm dubious that it is necessary to make such a big change so
close to release for hypothetical log statements. So, in the end, I agree
with you.

Yeah... As long as there is no problem yet.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Michael Paquier
michael.paquier@gmail.com
In reply to: Michael Paquier (#1)
Re: Use of RangeVar for partitioned tables in autovacuum

On Wed, Sep 27, 2017 at 11:28 AM, Michael Paquier
<michael.paquier@gmail.com> wrote:

In conclusion, I think that the open item of $subject should be
removed from the list, and we should try to get the multi-VACUUM patch
in to cover any future problems. I'll do so if there are no
objections.

And done. Moved to the non-bug section.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Amit Langote
Langote_Amit_f8@lab.ntt.co.jp
In reply to: Michael Paquier (#1)
Re: Use of RangeVar for partitioned tables in autovacuum

Thanks Michael for working on this.

On 2017/09/27 11:28, Michael Paquier wrote:

Hi all,

I have been looking more closely at the problem in $subject, that I
have mentioned a couple of times, like here:
/messages/by-id/CAB7nPqQa37OUne_rJzpmWC4EXQaLX9f27-Ma_-rSFL_3mj+HFQ@mail.gmail.com

As of HEAD, the RangeVar defined in calls of vacuum() is used for
error reporting, only in two cases: for autoanalyze and autovacuum
when reporting to users that a lock cannot be taken on a relation. The
thing is that autovacuum has the good idea to call vacuum() on only
one relation at the same time, with always a relation OID and a
RangeVar defined, so the code currently on HEAD is basically fine with
its error reporting, because get_rel_oids() will always work on the
relation OID with its RangeVar, and because code paths with manual
VACUUMs don't use the RangeVar for any reports.

Yeah, autovacuum_do_vac_analyze will never pass partitioned table OIDs to
vacuum, for example, because they're not RELKIND_RELATION relations.

While v10 is safe, I think that this is wrong in the long-term and
that may be a cause of bugs if people begin to generate reports for
manual VACUUMs, which is plausible in my opinion. The patch attached,
is what one solution would look like if we would like to make things
more robust as long as manual VACUUM commands can only specify one
relation at the same time. I have found that tracking the parent OID
by tweaking a bit get_rel_oids() was the most elegant solution.

The patch basically looks good to me, FWIW.

Please
note that this range of problems is something that I think is better
solved with the infrastructure that support for VACUUM with multiple
relations includes (last version here
/messages/by-id/766556DD-AA3C-42F7-ACF4-5DC97F41F151@amazon.com).
So I don't think that the patch attached should be integrated, still I
am attaching it to satisfy the curiosity of anybody looking at this
message.

Yeah, after looking at the code a bit, it seems that the problem won't
really occur for the case we're trying to apply this patch for.

In conclusion, I think that the open item of $subject should be
removed from the list, and we should try to get the multi-VACUUM patch
in to cover any future problems. I'll do so if there are no
objections.

+1 to this, taking the previous +1 back [1]/messages/by-id/8d810dd9-5f64-a5f3-c016-a81f05528fa8@lab.ntt.co.jp. :)

One comment on top of vacuum() is wrong by the way: in the case of a
manual VACUUM or ANALYZE, a NULL RangeVar is used if no relation is
specified in the command.

Yep, but it doesn't ever get accessed per our observations.

Thanks,
Amit

[1]: /messages/by-id/8d810dd9-5f64-a5f3-c016-a81f05528fa8@lab.ntt.co.jp
/messages/by-id/8d810dd9-5f64-a5f3-c016-a81f05528fa8@lab.ntt.co.jp

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers