PROC_VACUUM_FOR_WRAPAROUND doesn't work expectedly
I found autovacuum can be canceled by blocked backends even if the vacuum
is for preventing XID wraparound in 8.3.0 and HEAD. Autovacuum sets
PROC_VACUUM_FOR_WRAPAROUND flag just before vacuum, but the flag will be
cleared at the beginning of vacuum; PROC_VACUUM_FOR_WRAPAROUND is not set
during the vacuum.
The sequence is below:
vacuum()
-> CommitTransactionCommand()
-> ProcArrayEndTransaction()
-> proc->vacuumFlags &= ~PROC_VACUUM_STATE_MASK;
-> vacuum_rel()
PROC_VACUUM_STATE_MASK is defined as (0x0E), that is including
PROC_VACUUM_FOR_WRAPAROUND (0x08). The wraparound flag is cleared
before vacuum tasks.
I tried to make a patch to exclude PROC_VACUUM_FOR_WRAPAROUND
from PROC_VACUUM_STATE_MASK and make autovacuum workers to clear
PROC_VACUUM_FOR_WRAPAROUND by themselves. Is it a reasonable solution?
Index: src/backend/postmaster/autovacuum.c
===================================================================
--- src/backend/postmaster/autovacuum.c (HEAD)
+++ src/backend/postmaster/autovacuum.c (working copy)
@@ -2163,6 +2163,12 @@
PG_END_TRY();
/* the PGPROC flags are reset at the next end of transaction */
+ if (tab->at_wraparound)
+ {
+ LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
+ MyProc->vacuumFlags &= ~PROC_VACUUM_FOR_WRAPAROUND;
+ LWLockRelease(ProcArrayLock);
+ }
/* be tidy */
pfree(tab);
Index: src/include/storage/proc.h
===================================================================
--- src/include/storage/proc.h (HEAD)
+++ src/include/storage/proc.h (working copy)
@@ -45,7 +45,7 @@
#define PROC_VACUUM_FOR_WRAPAROUND 0x08 /* set by autovac only */
/* flags reset at EOXact */
-#define PROC_VACUUM_STATE_MASK (0x0E)
+#define PROC_VACUUM_STATE_MASK (PROC_IN_VACUUM | PROC_IN_ANALYZE)
/*
* Each backend has a PGPROC struct in shared memory. There is also a list of
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
On Fri, Mar 14, 2008 at 7:36 AM, ITAGAKI Takahiro
<itagaki.takahiro@oss.ntt.co.jp> wrote:
I tried to make a patch to exclude PROC_VACUUM_FOR_WRAPAROUND
from PROC_VACUUM_STATE_MASK and make autovacuum workers to clear
PROC_VACUUM_FOR_WRAPAROUND by themselves. Is it a reasonable solution?
Looks good to me. Otherwise we can pass additional parameter to
autovacuum_do_vac_analyze() and then use vacstmt to pass the information
to vacuum(). Not sure which is a cleaner way though.
I also noticed that inside autovacuum_do_vac_analyze(), we save the old
context (which is TopTransactionContext) and restore it back after vacuum()
returns. But vacuum() might have started a new transaction invalidating the
saved context. Do we see any problem here ?
Thanks,
Pavan
--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com
ITAGAKI Takahiro wrote:
I found autovacuum can be canceled by blocked backends even if the vacuum
is for preventing XID wraparound in 8.3.0 and HEAD. Autovacuum sets
PROC_VACUUM_FOR_WRAPAROUND flag just before vacuum, but the flag will be
cleared at the beginning of vacuum; PROC_VACUUM_FOR_WRAPAROUND is not set
during the vacuum.
Rats.
How about this other patch?
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Attachments:
vacuum-wraparound.patchtext/x-diff; charset=us-asciiDownload
*** src/backend/commands/vacuum.c 10 Mar 2008 02:04:08 -0000 1.366
--- src/backend/commands/vacuum.c 14 Mar 2008 12:54:14 -0000
***************
*** 1005,1011 ****
* which is probably Not Good.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
! MyProc->vacuumFlags |= PROC_IN_VACUUM;
LWLockRelease(ProcArrayLock);
}
--- 1005,1011 ----
* which is probably Not Good.
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
! MyProc->vacuumFlags |= PROC_IN_VACUUM | AutovacuumWraparoundFlag;
LWLockRelease(ProcArrayLock);
}
*** src/backend/postmaster/autovacuum.c 20 Feb 2008 14:01:45 -0000 1.72
--- src/backend/postmaster/autovacuum.c 14 Mar 2008 12:53:22 -0000
***************
*** 254,259 ****
--- 254,263 ----
/* PID of launcher, valid only in worker while shutting down */
int AutovacuumLauncherPid = 0;
+ /* Mask for PGPROC->vacuumFlags */
+ uint8 AutovacuumWraparoundFlag = 0;
+
+
#ifdef EXEC_BACKEND
static pid_t avlauncher_forkexec(void);
static pid_t avworker_forkexec(void);
***************
*** 2095,2107 ****
/* clean up memory before each iteration */
MemoryContextResetAndDeleteChildren(PortalContext);
! /* set the "vacuum for wraparound" flag in PGPROC */
if (tab->at_wraparound)
! {
! LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
! MyProc->vacuumFlags |= PROC_VACUUM_FOR_WRAPAROUND;
! LWLockRelease(ProcArrayLock);
! }
/*
* Save the relation name for a possible error message, to avoid a
--- 2099,2107 ----
/* clean up memory before each iteration */
MemoryContextResetAndDeleteChildren(PortalContext);
! /* set the "vacuum for wraparound" flag, to be set in PGPROC later */
if (tab->at_wraparound)
! AutovacuumWraparoundFlag = PROC_VACUUM_FOR_WRAPAROUND;
/*
* Save the relation name for a possible error message, to avoid a
***************
*** 2151,2157 ****
datname, nspname, relname);
EmitErrorReport();
! /* this resets the PGPROC flags too */
AbortOutOfAnyTransaction();
FlushErrorState();
MemoryContextResetAndDeleteChildren(PortalContext);
--- 2151,2161 ----
datname, nspname, relname);
EmitErrorReport();
! /*
! * Aborting the transaction resets the PGPROC flags; we need to
! * reset our own here
! */
! AutovacuumWraparoundFlag = 0;
AbortOutOfAnyTransaction();
FlushErrorState();
MemoryContextResetAndDeleteChildren(PortalContext);
***************
*** 2162,2168 ****
}
PG_END_TRY();
! /* the PGPROC flags are reset at the next end of transaction */
/* be tidy */
pfree(tab);
--- 2166,2176 ----
}
PG_END_TRY();
! /*
! * The PGPROC flags are reset at the next end of transaction;
! * reset our own
! */
! AutovacuumWraparoundFlag = 0;
/* be tidy */
pfree(tab);
*** src/include/postmaster/autovacuum.h 1 Jan 2008 19:45:58 -0000 1.14
--- src/include/postmaster/autovacuum.h 14 Mar 2008 12:53:55 -0000
***************
*** 60,63 ****
--- 60,66 ----
extern Size AutoVacuumShmemSize(void);
extern void AutoVacuumShmemInit(void);
+ /* PGPROC flag */
+ extern uint8 AutovacuumWraparoundFlag;
+
#endif /* AUTOVACUUM_H */
Alvaro Herrera <alvherre@commandprompt.com> writes:
How about this other patch?
That's really, really ugly. I'd rather see the flag passed down to
vacuum_rel as a regular function argument. I realize you'll need
to touch the signatures of a couple of levels of functions to do that,
but a global variable for this seems just dangerous.
regards, tom lane
Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
How about this other patch?
That's really, really ugly. I'd rather see the flag passed down to
vacuum_rel as a regular function argument. I realize you'll need
to touch the signatures of a couple of levels of functions to do that,
but a global variable for this seems just dangerous.
Okay, I'll do that instead. If I do it quickly, will it be present in
8.3.1? I think it was already tagged so my guess is it won't.
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alvaro Herrera <alvherre@commandprompt.com> writes:
Okay, I'll do that instead. If I do it quickly, will it be present in
8.3.1? I think it was already tagged so my guess is it won't.
I think Marc is planning to rebundle this evening, so if you can get
it done in the next few hours...
regards, tom lane
Alvaro Herrera wrote:
Tom Lane wrote:
That's really, really ugly. I'd rather see the flag passed down to
vacuum_rel as a regular function argument. I realize you'll need
to touch the signatures of a couple of levels of functions to do that,
but a global variable for this seems just dangerous.Okay, I'll do that instead.
Does this look better?
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
Attachments:
vacuum-wraparound-2.patchtext/x-diff; charset=us-asciiDownload
Index: src/backend/commands/vacuum.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/commands/vacuum.c,v
retrieving revision 1.364
diff -c -p -r1.364 vacuum.c
*** src/backend/commands/vacuum.c 11 Feb 2008 19:14:30 -0000 1.364
--- src/backend/commands/vacuum.c 14 Mar 2008 16:26:08 -0000
*************** static BufferAccessStrategy vac_strategy
*** 208,214 ****
static List *get_rel_oids(List *relids, const RangeVar *vacrel,
const char *stmttype);
static void vac_truncate_clog(TransactionId frozenXID);
! static void vacuum_rel(Oid relid, VacuumStmt *vacstmt, char expected_relkind);
static void full_vacuum_rel(Relation onerel, VacuumStmt *vacstmt);
static void scan_heap(VRelStats *vacrelstats, Relation onerel,
VacPageList vacuum_pages, VacPageList fraged_pages);
--- 208,215 ----
static List *get_rel_oids(List *relids, const RangeVar *vacrel,
const char *stmttype);
static void vac_truncate_clog(TransactionId frozenXID);
! static void vacuum_rel(Oid relid, VacuumStmt *vacstmt, char expected_relkind,
! bool for_wraparound);
static void full_vacuum_rel(Relation onerel, VacuumStmt *vacstmt);
static void scan_heap(VRelStats *vacrelstats, Relation onerel,
VacPageList vacuum_pages, VacPageList fraged_pages);
*************** static Size PageGetFreeSpaceWithFillFact
*** 262,267 ****
--- 263,271 ----
* relation OIDs to be processed, and vacstmt->relation is ignored.
* (The non-NIL case is currently only used by autovacuum.)
*
+ * for_wraparound is used by autovacuum to let us know when it's forcing
+ * a vacuum for wraparound, which should not be auto-cancelled.
+ *
* bstrategy is normally given as NULL, but in autovacuum it can be passed
* in to use the same buffer strategy object across multiple vacuum() calls.
*
*************** static Size PageGetFreeSpaceWithFillFact
*** 273,279 ****
*/
void
vacuum(VacuumStmt *vacstmt, List *relids,
! BufferAccessStrategy bstrategy, bool isTopLevel)
{
const char *stmttype = vacstmt->vacuum ? "VACUUM" : "ANALYZE";
volatile MemoryContext anl_context = NULL;
--- 277,283 ----
*/
void
vacuum(VacuumStmt *vacstmt, List *relids,
! BufferAccessStrategy bstrategy, bool for_wraparound, bool isTopLevel)
{
const char *stmttype = vacstmt->vacuum ? "VACUUM" : "ANALYZE";
volatile MemoryContext anl_context = NULL;
*************** vacuum(VacuumStmt *vacstmt, List *relids
*** 420,426 ****
Oid relid = lfirst_oid(cur);
if (vacstmt->vacuum)
! vacuum_rel(relid, vacstmt, RELKIND_RELATION);
if (vacstmt->analyze)
{
--- 424,430 ----
Oid relid = lfirst_oid(cur);
if (vacstmt->vacuum)
! vacuum_rel(relid, vacstmt, RELKIND_RELATION, for_wraparound);
if (vacstmt->analyze)
{
*************** vac_truncate_clog(TransactionId frozenXI
*** 965,971 ****
* At entry and exit, we are not inside a transaction.
*/
static void
! vacuum_rel(Oid relid, VacuumStmt *vacstmt, char expected_relkind)
{
LOCKMODE lmode;
Relation onerel;
--- 969,976 ----
* At entry and exit, we are not inside a transaction.
*/
static void
! vacuum_rel(Oid relid, VacuumStmt *vacstmt, char expected_relkind,
! bool for_wraparound)
{
LOCKMODE lmode;
Relation onerel;
*************** vacuum_rel(Oid relid, VacuumStmt *vacstm
*** 998,1003 ****
--- 1003,1012 ----
* contents of other tables is arguably broken, but we won't break it
* here by violating transaction semantics.)
*
+ * We also set the VACUUM_FOR_WRAPAROUND flag, which is passed down
+ * by autovacuum; it's used to avoid cancelling a vacuum that was
+ * invoked in an emergency.
+ *
* Note: this flag remains set until CommitTransaction or
* AbortTransaction. We don't want to clear it until we reset
* MyProc->xid/xmin, else OldestXmin might appear to go backwards,
*************** vacuum_rel(Oid relid, VacuumStmt *vacstm
*** 1005,1010 ****
--- 1014,1021 ----
*/
LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
MyProc->vacuumFlags |= PROC_IN_VACUUM;
+ if (for_wraparound)
+ MyProc->vacuumFlags |= PROC_VACUUM_FOR_WRAPAROUND;
LWLockRelease(ProcArrayLock);
}
*************** vacuum_rel(Oid relid, VacuumStmt *vacstm
*** 1137,1143 ****
* totally unimportant for toast relations.
*/
if (toast_relid != InvalidOid)
! vacuum_rel(toast_relid, vacstmt, RELKIND_TOASTVALUE);
/*
* Now release the session-level lock on the master table.
--- 1148,1154 ----
* totally unimportant for toast relations.
*/
if (toast_relid != InvalidOid)
! vacuum_rel(toast_relid, vacstmt, RELKIND_TOASTVALUE, for_wraparound);
/*
* Now release the session-level lock on the master table.
Index: src/backend/postmaster/autovacuum.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.71.2.1
diff -c -p -r1.71.2.1 autovacuum.c
*** src/backend/postmaster/autovacuum.c 20 Feb 2008 16:48:12 -0000 1.71.2.1
--- src/backend/postmaster/autovacuum.c 14 Mar 2008 16:12:36 -0000
*************** static void relation_needs_vacanalyze(Oi
*** 285,290 ****
--- 285,291 ----
static void autovacuum_do_vac_analyze(Oid relid, bool dovacuum,
bool doanalyze, int freeze_min_age,
+ bool for_wraparound,
BufferAccessStrategy bstrategy);
static HeapTuple get_pg_autovacuum_tuple_relid(Relation avRel, Oid relid);
static PgStat_StatTabEntry *get_pgstat_tabentry_relid(Oid relid, bool isshared,
*************** do_autovacuum(void)
*** 2095,2108 ****
/* clean up memory before each iteration */
MemoryContextResetAndDeleteChildren(PortalContext);
- /* set the "vacuum for wraparound" flag in PGPROC */
- if (tab->at_wraparound)
- {
- LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE);
- MyProc->vacuumFlags |= PROC_VACUUM_FOR_WRAPAROUND;
- LWLockRelease(ProcArrayLock);
- }
-
/*
* Save the relation name for a possible error message, to avoid a
* catalog lookup in case of an error. Note: they must live in a
--- 2096,2101 ----
*************** do_autovacuum(void)
*** 2126,2131 ****
--- 2119,2125 ----
tab->at_dovacuum,
tab->at_doanalyze,
tab->at_freeze_min_age,
+ tab->at_wraparound,
bstrategy);
/*
*************** relation_needs_vacanalyze(Oid relid,
*** 2604,2610 ****
*/
static void
autovacuum_do_vac_analyze(Oid relid, bool dovacuum, bool doanalyze,
! int freeze_min_age,
BufferAccessStrategy bstrategy)
{
VacuumStmt vacstmt;
--- 2598,2604 ----
*/
static void
autovacuum_do_vac_analyze(Oid relid, bool dovacuum, bool doanalyze,
! int freeze_min_age, bool for_wraparound,
BufferAccessStrategy bstrategy)
{
VacuumStmt vacstmt;
*************** autovacuum_do_vac_analyze(Oid relid, boo
*** 2631,2637 ****
/* Let pgstat know what we're doing */
autovac_report_activity(&vacstmt, relid);
! vacuum(&vacstmt, list_make1_oid(relid), bstrategy, true);
MemoryContextSwitchTo(old_cxt);
}
--- 2625,2631 ----
/* Let pgstat know what we're doing */
autovac_report_activity(&vacstmt, relid);
! vacuum(&vacstmt, list_make1_oid(relid), bstrategy, for_wraparound, true);
MemoryContextSwitchTo(old_cxt);
}
Index: src/backend/tcop/utility.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/tcop/utility.c,v
retrieving revision 1.289
diff -c -p -r1.289 utility.c
*** src/backend/tcop/utility.c 1 Jan 2008 19:45:52 -0000 1.289
--- src/backend/tcop/utility.c 14 Mar 2008 16:13:21 -0000
*************** ProcessUtility(Node *parsetree,
*** 1032,1038 ****
break;
case T_VacuumStmt:
! vacuum((VacuumStmt *) parsetree, NIL, NULL, isTopLevel);
break;
case T_ExplainStmt:
--- 1032,1038 ----
break;
case T_VacuumStmt:
! vacuum((VacuumStmt *) parsetree, NIL, NULL, false, isTopLevel);
break;
case T_ExplainStmt:
Index: src/include/commands/vacuum.h
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/include/commands/vacuum.h,v
retrieving revision 1.75
diff -c -p -r1.75 vacuum.h
*** src/include/commands/vacuum.h 1 Jan 2008 19:45:57 -0000 1.75
--- src/include/commands/vacuum.h 14 Mar 2008 16:13:40 -0000
*************** extern int vacuum_freeze_min_age;
*** 114,120 ****
/* in commands/vacuum.c */
extern void vacuum(VacuumStmt *vacstmt, List *relids,
! BufferAccessStrategy bstrategy, bool isTopLevel);
extern void vac_open_indexes(Relation relation, LOCKMODE lockmode,
int *nindexes, Relation **Irel);
extern void vac_close_indexes(int nindexes, Relation *Irel, LOCKMODE lockmode);
--- 114,120 ----
/* in commands/vacuum.c */
extern void vacuum(VacuumStmt *vacstmt, List *relids,
! BufferAccessStrategy bstrategy, bool for_wraparound, bool isTopLevel);
extern void vac_open_indexes(Relation relation, LOCKMODE lockmode,
int *nindexes, Relation **Irel);
extern void vac_close_indexes(int nindexes, Relation *Irel, LOCKMODE lockmode);
Alvaro Herrera <alvherre@commandprompt.com> writes:
Does this look better?
Yeah, works for me. Please apply.
regards, tom lane
Tom Lane wrote:
Alvaro Herrera <alvherre@commandprompt.com> writes:
Does this look better?
Yeah, works for me. Please apply.
Applied to HEAD and 8.3.
Thanks, Takahiro-san, for the report!
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
"Pavan Deolasee" <pavan.deolasee@gmail.com> writes:
I also noticed that inside autovacuum_do_vac_analyze(), we save the old
context (which is TopTransactionContext) and restore it back after vacuum()
returns. But vacuum() might have started a new transaction invalidating the
saved context. Do we see any problem here ?
I agree, that looks pretty darn bogus. The other problem with it is
that it's running vacuum() in an indefinite-lifespan context. Perhaps
that has something to do with the report we saw awhile back of autovac
leaking memory ...
regards, tom lane
Tom Lane escribi�:
"Pavan Deolasee" <pavan.deolasee@gmail.com> writes:
I also noticed that inside autovacuum_do_vac_analyze(), we save the old
context (which is TopTransactionContext) and restore it back after vacuum()
returns. But vacuum() might have started a new transaction invalidating the
saved context. Do we see any problem here ?I agree, that looks pretty darn bogus.
Sorry, I failed to notice this part of Pavan's reply. Thanks for
patching it.
The other problem with it is that it's running vacuum() in an
indefinite-lifespan context. Perhaps that has something to do with
the report we saw awhile back of autovac leaking memory ...
Hmm, I'm not sure which memory leak are you referring to, but if it's
the same I'm thinking of, then it cannot be the same because this one
occurs on the worker and the other was on the launcher; also, I patched
that one:
----------------------------
revision 1.58
date: 2007-09-12 18:14:59 -0400; author: alvherre; state: Exp; lines: +20 -3;Fix a memory leak in the autovacuum launcher code. Noted by Darcy Buskermolen,
who reported it privately to me.
----------------------------
Hmm, well, if he reported it privately to me then you cannot have heard
of it, right? So perhaps it is indeed a different one ...
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alvaro Herrera <alvherre@commandprompt.com> writes:
Tom Lane escribi�:
The other problem with it is that it's running vacuum() in an
indefinite-lifespan context. Perhaps that has something to do with
the report we saw awhile back of autovac leaking memory ...
Hmm, I'm not sure which memory leak are you referring to, but if it's
the same I'm thinking of, then it cannot be the same because this one
occurs on the worker and the other was on the launcher; also, I patched
that one:
I was thinking of Erik Jones' report of TopMemoryContext bloat in a
database with 200000 tables (in pre-8.3 code). But I guess this still
doesn't fit, because any leakage induced by vacuum() would have been in
AutovacMemCxt, and that wasn't what he saw. So I still don't know what
was happening with Erik's issue.
regards, tom lane