BUG #3326: Invalid lower bound of autovacuum_cost_limit

Started by Galy Leeover 18 years ago10 messages
#1Galy Lee
lee.galy@oss.ntt.co.jp

The following bug has been logged online:

Bug reference: 3326
Logged by: Galy Lee
Email address: lee.galy@oss.ntt.co.jp
PostgreSQL version: 8.3
Operating system: Red Hat 4.
Description: Invalid lower bound of autovacuum_cost_limit
Details:

Hello

I found some bugs which relative to the autovacuum_cost_limit GUC
parameter.

* Bug-1: Invalid lower bound of autovacuum_cost_limit

autovacuum_vacuum_cost_limit should be the following value:
autovacuum_vacuum_cost_limit = -1, or [1, 10000]
(0 should be prohibited. )

But 0 can also be accepted for autovacuum_vacuum_cost_limit now.

This causes zero-division error for autovacuum:

ERROR: floating-point exception
DETAIL: An invalid floating-point operation was signaled. This
probably means an out-of-range result or an invalid operation,
such as division by zero.

* Bug-2: 0-cost-limit for autovacuum worker

When autovacuum_max_workers > autovacuum_vacuum_cost_limit, the above
zero-division error also happened.

* Bug-3: no GUC constrain check for pg_autovacuum

The settings in pg_autovacuum are not checked enough now. Invalid value
can be passed to autovacuum, this also causes some columns in
pg_autovacuum has inconsistent upper and lower bound with their original
GUC constrain.

pg_autovacuum colum | definition | GUC constrain
--------------------+------------+--------------------------
vac_base_thresh | integer | [0, INT_MAX]
vac_scale_factor | real | [0.0, 100.0]
anl_base_thresh | integer | [0, INT_MAX]
anl_scale_factor | real | [0.0, 100.0]
vac_cost_delay | integer | [-1, 1000]
vac_cost_limit | integer | [-1, 10000]
freeze_min_age | integer | [0, 1000000000]
freeze_max_age | integer | [100000000, 2000000000]

The above table shows the wrong mapping between pg_autovacuum columns
and GUC constrain.

#2Alvaro Herrera
alvherre@commandprompt.com
In reply to: Galy Lee (#1)
Re: BUG #3326: Invalid lower bound of autovacuum_cost_limit

Galy Lee wrote:

Hi Galy,

The following bug has been logged online:

Bug reference: 3326
Logged by: Galy Lee
Email address: lee.galy@oss.ntt.co.jp
PostgreSQL version: 8.3
Operating system: Red Hat 4.
Description: Invalid lower bound of autovacuum_cost_limit
Details:

Damn. Thanks for reporting -- I'll investigate these issues.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

#3Alvaro Herrera
alvherre@commandprompt.com
In reply to: Galy Lee (#1)
1 attachment(s)
Re: [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Galy Lee wrote:

Hi,

I'll deal with each issue separately.

* Bug-1: Invalid lower bound of autovacuum_cost_limit

autovacuum_vacuum_cost_limit should be the following value:
autovacuum_vacuum_cost_limit = -1, or [1, 10000]
(0 should be prohibited. )

But 0 can also be accepted for autovacuum_vacuum_cost_limit now.

This is solved easily by adding a check in an assign hook (attached).
The only remaining problem here is that other error messages could be
clearer.

So this is correct:

$ postmaster -c autovacuum_vacuum_cost_limit=0
17902 FATAL: invalid value for parameter "autovacuum_vacuum_cost_limit": 0

But this is misleading (started postmaster with good value, then edited
postgresql.conf and entered "-2"):

17903 LOG: received SIGHUP, reloading configuration files
17903 LOG: -2 is outside the valid range for parameter "autovacuum_vacuum_cost_limit" (-1 .. 1000)

Note how it still says the range is -1 .. 1000.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Attachments:

autovac-cost-limit-2.patchtext/x-diff; charset=us-asciiDownload
Index: src/backend/utils/misc/guc.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/utils/misc/guc.c,v
retrieving revision 1.395
diff -c -p -r1.395 guc.c
*** src/backend/utils/misc/guc.c	5 Jun 2007 21:50:19 -0000	1.395
--- src/backend/utils/misc/guc.c	7 Jun 2007 15:51:55 -0000
*************** static const char *show_tcp_keepalives_i
*** 169,174 ****
--- 169,175 ----
  static const char *show_tcp_keepalives_count(void);
  static bool assign_autovacuum_max_workers(int newval, bool doit, GucSource source);
  static bool assign_maxconnections(int newval, bool doit, GucSource source);
+ static bool assign_autovac_vac_cost_limit(int newval, bool doit, GucSource source);
  
  /*
   * GUC option variables that are exported from this module
*************** static struct config_int ConfigureNamesI
*** 1329,1335 ****
  			NULL
  		},
  		&autovacuum_vac_cost_limit,
! 		-1, -1, 10000, NULL, NULL
  	},
  
  	{
--- 1330,1336 ----
  			NULL
  		},
  		&autovacuum_vac_cost_limit,
! 		-1, -1, 10000, &assign_autovac_vac_cost_limit, NULL
  	},
  
  	{
*************** assign_autovacuum_max_workers(int newval
*** 6831,6834 ****
--- 6832,6842 ----
  	return true;
  }
  
+ /* autovacuum_vacuum_cost_limit may be -1, but not 0 */
+ static bool
+ assign_autovac_vac_cost_limit(int newval, bool doit, GucSource source)
+ {
+ 	return newval != 0;
+ }
+ 
  #include "guc-file.c"
#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#3)
Re: [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Alvaro Herrera <alvherre@commandprompt.com> writes:

But this is misleading (started postmaster with good value, then edited
postgresql.conf and entered "-2"):

17903 LOG: received SIGHUP, reloading configuration files
17903 LOG: -2 is outside the valid range for parameter "autovacuum_vacuum_cost_limit" (-1 .. 1000)

Note how it still says the range is -1 .. 1000.

Can we redefine things to make zero be the "disabled" value, thus
keeping the range of valid values contiguous?

regards, tom lane

#5Alvaro Herrera
alvherre@commandprompt.com
In reply to: Tom Lane (#4)
Re: [PATCHES] [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Tom Lane wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

But this is misleading (started postmaster with good value, then edited
postgresql.conf and entered "-2"):

17903 LOG: received SIGHUP, reloading configuration files
17903 LOG: -2 is outside the valid range for parameter "autovacuum_vacuum_cost_limit" (-1 .. 1000)

Note how it still says the range is -1 .. 1000.

Can we redefine things to make zero be the "disabled" value, thus
keeping the range of valid values contiguous?

That would be another solution ... though it would be different from the
valid value for autovacuum_vacuum_cost_delay (on which 0 is a valid
value). Also it would be a different value from previous versions.

I don't think either of these is a showstopper, so let's go for that if
nobody objects.

--
Alvaro Herrera Developer, http://www.PostgreSQL.org/
<inflex> really, I see PHP as like a strange amalgamation of C, Perl, Shell
<crab> inflex: you know that "amalgam" means "mixture with mercury",
more or less, right?
<crab> i.e., "deadly poison"

#6Matthew T. O'Connor
matthew@zeut.net
In reply to: Alvaro Herrera (#5)
Re: [PATCHES] [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Alvaro Herrera wrote:

Tom Lane wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

But this is misleading (started postmaster with good value, then edited
postgresql.conf and entered "-2"):
17903 LOG: received SIGHUP, reloading configuration files
17903 LOG: -2 is outside the valid range for parameter "autovacuum_vacuum_cost_limit" (-1 .. 1000)
Note how it still says the range is -1 .. 1000.

Can we redefine things to make zero be the "disabled" value, thus
keeping the range of valid values contiguous?

That would be another solution ... though it would be different from the
valid value for autovacuum_vacuum_cost_delay (on which 0 is a valid
value). Also it would be a different value from previous versions.

I don't think either of these is a showstopper, so let's go for that if
nobody objects.

Can you make 0 and -1 both valid disabled values? That way it will be
compatible with previous releases.

#7Alvaro Herrera
alvherre@commandprompt.com
In reply to: Matthew T. O'Connor (#6)
Re: [PATCHES] [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Matthew T. O'Connor wrote:

Alvaro Herrera wrote:

Tom Lane wrote:

Alvaro Herrera <alvherre@commandprompt.com> writes:

But this is misleading (started postmaster with good value, then edited
postgresql.conf and entered "-2"):
17903 LOG: received SIGHUP, reloading configuration files
17903 LOG: -2 is outside the valid range for parameter
"autovacuum_vacuum_cost_limit" (-1 .. 1000)
Note how it still says the range is -1 .. 1000.

Can we redefine things to make zero be the "disabled" value, thus
keeping the range of valid values contiguous?

That would be another solution ... though it would be different from the
valid value for autovacuum_vacuum_cost_delay (on which 0 is a valid
value). Also it would be a different value from previous versions.

I don't think either of these is a showstopper, so let's go for that if
nobody objects.

Can you make 0 and -1 both valid disabled values? That way it will be
compatible with previous releases.

Heh, sure, we can do that too and it doesn't seem like anybody would
object. I will patch the documentation so that that the "disabled"
value is zero, and still allow -1. That way it doesn't seem like there
should be any objection.

--
Alvaro Herrera http://www.flickr.com/photos/alvherre/
"Escucha y olvidar�s; ve y recordar�s; haz y entender�s" (Confucio)

#8Alvaro Herrera
alvherre@commandprompt.com
In reply to: Alvaro Herrera (#7)
1 attachment(s)
Re: [PATCHES] [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Alvaro Herrera wrote:

Matthew T. O'Connor wrote:

Can you make 0 and -1 both valid disabled values? That way it will be
compatible with previous releases.

Heh, sure, we can do that too and it doesn't seem like anybody would
object. I will patch the documentation so that that the "disabled"
value is zero, and still allow -1. That way it doesn't seem like there
should be any objection.

Patch attached.

--
Alvaro Herrera http://www.amazon.com/gp/registry/5ZYLFMCVHXC
"Saca el libro que tu religi�n considere como el indicado para encontrar la
oraci�n que traiga paz a tu alma. Luego rebootea el computador
y ve si funciona" (Carlos Ducl�s)

Attachments:

autovac-zero-cost-limit.patchtext/x-diff; charset=us-asciiDownload
Index: doc/src/sgml/catalogs.sgml
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/doc/src/sgml/catalogs.sgml,v
retrieving revision 2.153
diff -c -p -r2.153 catalogs.sgml
*** doc/src/sgml/catalogs.sgml	5 Jun 2007 21:31:03 -0000	2.153
--- doc/src/sgml/catalogs.sgml	8 Jun 2007 14:40:10 -0000
***************
*** 1339,1347 ****
     be used for this particular value.  Observe that the
     <structfield>vac_cost_delay</> variable inherits its default value from the
     <xref linkend="guc-autovacuum-vacuum-cost-delay"> configuration parameter,
!    or from <varname>vacuum_cost_delay</> if the former is set to a negative
!    value.  The same applies to <structfield>vac_cost_limit</>.
!    Also, autovacuum will ignore attempts to set a per-table
     <structfield>freeze_max_age</> larger than the system-wide setting (it can only be set
     smaller), and the <structfield>freeze_min_age value</> will be limited to half the
     system-wide <xref linkend="guc-autovacuum-freeze-max-age"> setting.
--- 1339,1350 ----
     be used for this particular value.  Observe that the
     <structfield>vac_cost_delay</> variable inherits its default value from the
     <xref linkend="guc-autovacuum-vacuum-cost-delay"> configuration parameter,
!    or from <xref linkend="guc-vacuum-cost-delay"> if the former is set to a negative
!    value.  <structfield>vac_cost_limit</> is an exception to this rule, because
!    the value <literal>0</> is used to indicate that the
!    <xref linkend="guc-autovacuum-vacuum-cost-limit"> configuration parameter
!    should be used, or <xref linkend="guc-vacuum-cost-limit"> if the former is set to a
!    zero or negative value.  Note that autovacuum will ignore attempts to set a per-table
     <structfield>freeze_max_age</> larger than the system-wide setting (it can only be set
     smaller), and the <structfield>freeze_min_age value</> will be limited to half the
     system-wide <xref linkend="guc-autovacuum-freeze-max-age"> setting.
Index: doc/src/sgml/config.sgml
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/doc/src/sgml/config.sgml,v
retrieving revision 1.126
diff -c -p -r1.126 config.sgml
*** doc/src/sgml/config.sgml	7 Jun 2007 19:19:56 -0000	1.126
--- doc/src/sgml/config.sgml	8 Jun 2007 14:15:33 -0000
*************** SELECT * FROM parent WHERE key = 2400;
*** 3356,3362 ****
        <listitem>
         <para>
          Specifies the cost limit value that will be used in automatic
!         <command>VACUUM</> operations.  If <literal>-1</> is specified (which is the
          default), the regular
          <xref linkend="guc-vacuum-cost-limit"> value will be used.  Note that
          the value is distributed proportionally among the running autovacuum
--- 3356,3362 ----
        <listitem>
         <para>
          Specifies the cost limit value that will be used in automatic
!         <command>VACUUM</> operations.  If <literal>0</> is specified (which is the
          default), the regular
          <xref linkend="guc-vacuum-cost-limit"> value will be used.  Note that
          the value is distributed proportionally among the running autovacuum
Index: src/backend/postmaster/autovacuum.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.47
diff -c -p -r1.47 autovacuum.c
*** src/backend/postmaster/autovacuum.c	30 May 2007 20:11:57 -0000	1.47
--- src/backend/postmaster/autovacuum.c	8 Jun 2007 14:23:35 -0000
*************** static void
*** 1548,1554 ****
  autovac_balance_cost(void)
  {
  	WorkerInfo	worker;
! 	int         vac_cost_limit = (autovacuum_vac_cost_limit >= 0 ?
  								  autovacuum_vac_cost_limit : VacuumCostLimit);
  	int         vac_cost_delay = (autovacuum_vac_cost_delay >= 0 ?
  								  autovacuum_vac_cost_delay : VacuumCostDelay);
--- 1548,1554 ----
  autovac_balance_cost(void)
  {
  	WorkerInfo	worker;
! 	int         vac_cost_limit = (autovacuum_vac_cost_limit > 0 ?
  								  autovacuum_vac_cost_limit : VacuumCostLimit);
  	int         vac_cost_delay = (autovacuum_vac_cost_delay >= 0 ?
  								  autovacuum_vac_cost_delay : VacuumCostDelay);
*************** table_recheck_autovac(Oid relid)
*** 2143,2151 ****
  		 */
  		if (avForm != NULL)
  		{
! 			vac_cost_limit = (avForm->vac_cost_limit >= 0) ?
  				avForm->vac_cost_limit :
! 				((autovacuum_vac_cost_limit >= 0) ?
  				 autovacuum_vac_cost_limit : VacuumCostLimit);
  
  			vac_cost_delay = (avForm->vac_cost_delay >= 0) ?
--- 2143,2151 ----
  		 */
  		if (avForm != NULL)
  		{
! 			vac_cost_limit = (avForm->vac_cost_limit > 0) ?
  				avForm->vac_cost_limit :
! 				((autovacuum_vac_cost_limit > 0) ?
  				 autovacuum_vac_cost_limit : VacuumCostLimit);
  
  			vac_cost_delay = (avForm->vac_cost_delay >= 0) ?
*************** table_recheck_autovac(Oid relid)
*** 2158,2164 ****
  		}
  		else
  		{
! 			vac_cost_limit = (autovacuum_vac_cost_limit >= 0) ?
  				autovacuum_vac_cost_limit : VacuumCostLimit;
  
  			vac_cost_delay = (autovacuum_vac_cost_delay >= 0) ?
--- 2158,2164 ----
  		}
  		else
  		{
! 			vac_cost_limit = (autovacuum_vac_cost_limit > 0) ?
  				autovacuum_vac_cost_limit : VacuumCostLimit;
  
  			vac_cost_delay = (autovacuum_vac_cost_delay >= 0) ?
Index: src/backend/utils/misc/guc.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/utils/misc/guc.c,v
retrieving revision 1.395
diff -c -p -r1.395 guc.c
*** src/backend/utils/misc/guc.c	5 Jun 2007 21:50:19 -0000	1.395
--- src/backend/utils/misc/guc.c	8 Jun 2007 14:12:49 -0000
*************** static struct config_int ConfigureNamesI
*** 1329,1335 ****
  			NULL
  		},
  		&autovacuum_vac_cost_limit,
! 		-1, -1, 10000, NULL, NULL
  	},
  
  	{
--- 1329,1335 ----
  			NULL
  		},
  		&autovacuum_vac_cost_limit,
! 		0, -1, 10000, NULL, NULL
  	},
  
  	{
Index: src/backend/utils/misc/postgresql.conf.sample
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/utils/misc/postgresql.conf.sample,v
retrieving revision 1.216
diff -c -p -r1.216 postgresql.conf.sample
*** src/backend/utils/misc/postgresql.conf.sample	3 Jun 2007 17:08:15 -0000	1.216
--- src/backend/utils/misc/postgresql.conf.sample	8 Jun 2007 14:25:11 -0000
***************
*** 394,401 ****
  #autovacuum_vacuum_cost_delay = -1	# default vacuum cost delay for 
  					# autovacuum, -1 means use 
  					# vacuum_cost_delay
! #autovacuum_vacuum_cost_limit = -1	# default vacuum cost limit for 
! 					# autovacuum, -1 means use
  					# vacuum_cost_limit
  
  
--- 394,401 ----
  #autovacuum_vacuum_cost_delay = -1	# default vacuum cost delay for 
  					# autovacuum, -1 means use 
  					# vacuum_cost_delay
! #autovacuum_vacuum_cost_limit = 0	# default vacuum cost limit for 
! 					# autovacuum, 0 means use
  					# vacuum_cost_limit
  
  
#9Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#8)
Re: [PATCHES] [BUGS] BUG #3326: Invalid lower bound of autovacuum_cost_limit

Alvaro Herrera <alvherre@commandprompt.com> writes:

Matthew T. O'Connor wrote:

Can you make 0 and -1 both valid disabled values? That way it will be
compatible with previous releases.

Heh, sure, we can do that too and it doesn't seem like anybody would
object. I will patch the documentation so that that the "disabled"
value is zero, and still allow -1. That way it doesn't seem like there
should be any objection.

Patch attached.

It seems like documenting vac_cost_limit as being different from the
others will just create perceived complexity/confusion, with no real
benefit. I'd suggest leaving the documentation and the default value
alone, and applying just the part of the patch that causes 0 to be
silently treated as if it were -1.

A comment at the spot where this is done would be a good idea, but
I don't think we need to say anything in the SGML docs.

regards, tom lane

#10Alvaro Herrera
alvherre@commandprompt.com
In reply to: Galy Lee (#1)
1 attachment(s)
Re: BUG #3326: Invalid lower bound of autovacuum_cost_limit

Galy Lee wrote:

* Bug-2: 0-cost-limit for autovacuum worker

When autovacuum_max_workers > autovacuum_vacuum_cost_limit, the above
zero-division error also happened.

Ah, this is a problem in the balance code -- it fails to consider that
the cost limit may be end up being 0 in the integer calculations. This
patch fixes this problem.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Attachments:

autovac-balance-limit.patchtext/x-diff; charset=us-asciiDownload
Index: src/backend/postmaster/autovacuum.c
===================================================================
RCS file: /home/alvherre/Code/cvs/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.47
diff -c -p -r1.47 autovacuum.c
*** src/backend/postmaster/autovacuum.c	30 May 2007 20:11:57 -0000	1.47
--- src/backend/postmaster/autovacuum.c	8 Jun 2007 14:58:19 -0000
*************** autovac_balance_cost(void)
*** 1595,1601 ****
  			int     limit = (int)
  				(cost_avail * worker->wi_cost_limit_base / cost_total);
  
! 			worker->wi_cost_limit = Min(limit, worker->wi_cost_limit_base);
  
  			elog(DEBUG2, "autovac_balance_cost(pid=%u db=%u, rel=%u, cost_limit=%d, cost_delay=%d)",
  				 worker->wi_workerpid, worker->wi_dboid,
--- 1595,1605 ----
  			int     limit = (int)
  				(cost_avail * worker->wi_cost_limit_base / cost_total);
  
! 			/*
! 			 * We put a lower bound of 1 to the cost_limit, to avoid division-
! 			 * by-zero in the vacuum code.
! 			 */
! 			worker->wi_cost_limit = Max(Min(limit, worker->wi_cost_limit_base), 1);
  
  			elog(DEBUG2, "autovac_balance_cost(pid=%u db=%u, rel=%u, cost_limit=%d, cost_delay=%d)",
  				 worker->wi_workerpid, worker->wi_dboid,