Re: Auto-explain patch

Started by Dean Rasheedover 17 years ago61 messages
#1Dean Rasheed
dean_rasheed@hotmail.com
1 attachment(s)

Hi,

This is a small patch I posted a few months back, and then kinda forgot
about / got distracted with other things.

Is there any interest in this? If so I am willing to put more work into
it, if people like it or have suggested improvements. Otherwise I'll let it
drop.

Here's what is does:

As it stands, basically it's just another debug parameter, called
debug_explain_plan, similar to debug_print_plan, except that the
output is in the form of EXPLAIN ANALYSE.

The main advantage it offers over a standard EXPLAIN ANALYSE is that
it explains *all* SQL executed, including any from within stored
prodecures and triggers, so it is quite useful for debugging these.

With a suitable logging level, it can also be used to produce very
verbose logfile output to help spot any inefficient database access by
other applications.

Example usage:

test=# SET debug_explain_plan=on;
SET
test=# SET client_min_messages=debug1;
SET
test=# CREATE TABLE foo(a int primary key);
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "foo_pkey" for table "foo"
CREATE TABLE
test=# INSERT INTO foo VALUES (1), (2), (3), (4), (5);
DEBUG: ------------------- query plan -------------------
DETAIL: Values Scan on "*VALUES*" (cost=0.00..0.06 rows=5 width=4) (actual time=0.001..0.006 rows=5 loops=1)
Query runtime: 0.329 ms
INSERT 0 5
test=# CREATE FUNCTION foo() RETURNS int as 'select max(a) from foo;' LANGUAGE SQL STABLE;
CREATE FUNCTION
test=# SELECT * FROM foo WHERE a=foo();
DEBUG: ------------------- query plan -------------------
DETAIL: Result (cost=0.04..0.05 rows=1 width=0) (actual time=0.044..0.044 rows=1 loops=1)
InitPlan
-> Limit (cost=0.00..0.04 rows=1 width=4) (actual time=0.032..0.034 rows=1 loops=1)
-> Index Scan Backward using foo_pkey on foo (cost=0.00..84.25 rows=2400 width=4) (actual time=0.025..0.025 rows=1 loops=1)
Filter: (a IS NOT NULL)
Query runtime: 0.050 ms
CONTEXT: SQL function "foo" statement 1
DEBUG: ------------------- query plan -------------------
DETAIL: Result (cost=0.04..0.05 rows=1 width=0) (actual time=0.037..0.037 rows=1 loops=1)
InitPlan
-> Limit (cost=0.00..0.04 rows=1 width=4) (actual time=0.027..0.029 rows=1 loops=1)
-> Index Scan Backward using foo_pkey on foo (cost=0.00..84.25 rows=2400 width=4) (actual time=0.021..0.021 rows=1 loops=1)
Filter: (a IS NOT NULL)
Query runtime: 0.044 ms
CONTEXT: SQL function "foo" statement 1
DEBUG: ------------------- query plan -------------------
DETAIL: Index Scan using foo_pkey on foo (cost=0.25..8.52 rows=1 width=4) (actual time=1.638..1.642 rows=1 loops=1)
Index Cond: (a = foo())
Query runtime: 1.686 ms
a
---
5
(1 row)

test=# EXPLAIN SELECT * FROM foo WHERE a=foo();
DEBUG: ------------------- query plan -------------------
DETAIL: Result (cost=0.04..0.05 rows=1 width=0) (actual time=0.012..0.012 rows=1 loops=1)
InitPlan
-> Limit (cost=0.00..0.04 rows=1 width=4) (actual time=0.011..0.011 rows=1 loops=1)
-> Index Scan Backward using foo_pkey on foo (cost=0.00..84.25 rows=2400 width=4) (actual time=0.010..0.010 rows=1 loops=1)
Filter: (a IS NOT NULL)
Query runtime: 0.014 ms
CONTEXT: SQL function "foo" statement 1
QUERY PLAN
--------------------------------------------------------------------
Index Scan using foo_pkey on foo (cost=0.25..8.52 rows=1 width=4)
Index Cond: (a = foo())
(2 rows)

(The last example shows foo() being called during the planning of this
query, which explains why it is called twice during the previous execution)

Simon Riggs reviewed this last time and said that what this patch
currently does is probably not exactly what is wanted for PostgreSQL.
Possible improvements might be to integrate this with the EXPLAIN
command (eg. EXPLAIN TRACE query) and have a separate parameter
(log_explain) for logging purposes.

Comments?

Regards, Dean

_________________________________________________________________
Live Search Charades - guess correctly and find hidden videos
http://www.searchcharades.com/

Attachments:

auto-explain.patchtext/x-patchDownload
*** ./doc/src/sgml/config.sgml.orig	Sun Jan 27 19:12:28 2008
--- ./doc/src/sgml/config.sgml	Sun Jan 27 21:58:32 2008
***************
*** 2790,2795 ****
--- 2790,2796 ----
        <term><varname>debug_print_rewritten</varname> (<type>boolean</type>)</term>
        <term><varname>debug_print_plan</varname> (<type>boolean</type>)</term>
        <term><varname>debug_pretty_print</varname> (<type>boolean</type>)</term>
+       <term><varname>debug_explain_plan</varname> (<type>boolean</type>)</term>
        <indexterm>
         <primary><varname>debug_print_parse</> configuration parameter</primary>
        </indexterm>
***************
*** 2802,2807 ****
--- 2803,2811 ----
        <indexterm>
         <primary><varname>debug_pretty_print</> configuration parameter</primary>
        </indexterm>
+       <indexterm>
+        <primary><varname>debug_explain_plan</> configuration parameter</primary>
+       </indexterm>
        <listitem>
         <para>
          These parameters enable various debugging output to be emitted.
***************
*** 2809,2820 ****
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
        </listitem>
       </varlistentry>
  
--- 2813,2837 ----
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>debug_explain_plan</varname> prints
!         the plan for each executed query in the same format as
!         <command>EXPLAIN ANALYZE</>. This includes queries executed from
!         within functions.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
+ 
+        <note>
+         <para>
+          The reports produced by <varname>debug_explain_plan</varname>
+          are produced at a lower level in the database, as each query
+          is executed, including queries executed from functions, so
+          the output may be more verbose that of <command>EXPLAIN ANALYZE</>
+          and the timings may differ.
+         </para>
+        </note>
        </listitem>
       </varlistentry>
  
*** ./src/backend/commands/explain.c.orig	Tue Jan  1 19:45:50 2008
--- ./src/backend/commands/explain.c	Sat Jan 26 11:55:54 2008
***************
*** 39,65 ****
  /* Hook for plugins to get control in explain_get_index_name() */
  explain_get_index_name_hook_type explain_get_index_name_hook = NULL;
  
- 
- typedef struct ExplainState
- {
- 	/* options */
- 	bool		printNodes;		/* do nodeToString() too */
- 	bool		printAnalyze;	/* print actual times */
- 	/* other states */
- 	PlannedStmt *pstmt;			/* top of plan */
- 	List	   *rtable;			/* range table */
- } ExplainState;
- 
  static void ExplainOneQuery(Query *query, ExplainStmt *stmt,
  				const char *queryString,
  				ParamListInfo params, TupOutputState *tstate);
  static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
  				StringInfo buf);
- static double elapsed_time(instr_time *starttime);
- static void explain_outNode(StringInfo str,
- 				Plan *plan, PlanState *planstate,
- 				Plan *outer_plan,
- 				int indent, ExplainState *es);
  static void show_scan_qual(List *qual, const char *qlabel,
  			   int scanrelid, Plan *outer_plan, Plan *inner_plan,
  			   StringInfo str, int indent, ExplainState *es);
--- 39,49 ----
***************
*** 402,408 ****
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! static double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
--- 386,392 ----
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
***************
*** 436,442 ****
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! static void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
--- 420,426 ----
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
*** ./src/backend/executor/execMain.c.orig	Tue Jan  1 19:45:50 2008
--- ./src/backend/executor/execMain.c	Sat Jan 26 14:34:26 2008
***************
*** 39,44 ****
--- 39,45 ----
  #include "catalog/heap.h"
  #include "catalog/namespace.h"
  #include "catalog/toasting.h"
+ #include "commands/explain.h"
  #include "commands/tablespace.h"
  #include "commands/trigger.h"
  #include "executor/execdebug.h"
***************
*** 52,57 ****
--- 53,59 ----
  #include "utils/acl.h"
  #include "utils/lsyscache.h"
  #include "utils/memutils.h"
+ #include "utils/guc.h"
  
  
  typedef struct evalPlanQual
***************
*** 185,190 ****
--- 187,198 ----
  	}
  
  	/*
+ 	 * If we are explaining all queries, enable instrumentation
+ 	 */
+ 	if (Debug_explain_plan)
+ 		queryDesc->doInstrument = true;
+ 
+ 	/*
  	 * Copy other important information into the EState
  	 */
  	estate->es_snapshot = queryDesc->snapshot;
***************
*** 227,232 ****
--- 235,243 ----
  	bool		sendTuples;
  	TupleTableSlot *result;
  	MemoryContext oldcontext;
+ 	bool		explainPlan = Debug_explain_plan;
+ 	instr_time	starttime;
+ 	double		totaltime;
  
  	/* sanity checks */
  	Assert(queryDesc != NULL);
***************
*** 236,241 ****
--- 247,263 ----
  	Assert(estate != NULL);
  
  	/*
+ 	 * If explaining all queries, record the start time before running
+ 	 * the query. This is not quite the same as EXPLAIN ANALYSE, which
+ 	 * starts timing before ExecutorStart(). Here we are only timing
+ 	 * how long the query takes to run, once initialised and also
+ 	 * excluding any triggers (which may themselves run queries which
+ 	 * will be instrumented separately).
+ 	 */
+ 	if (explainPlan)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/*
  	 * Switch into per-query memory context
  	 */
  	oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
***************
*** 278,283 ****
--- 300,340 ----
  		(*dest->rShutdown) (dest);
  
  	MemoryContextSwitchTo(oldcontext);
+ 
+ 	/*
+ 	 * If explaining all queries, log the execution plan using
+ 	 * the same format as EXPLAIN ANALYSE
+ 	 */
+ 	if (explainPlan)
+ 	{
+ 		ExplainState *es;
+ 		StringInfoData buf;
+ 
+ 		totaltime = elapsed_time(&starttime);
+ 
+ 		es = (ExplainState *) palloc0(sizeof(ExplainState));
+ 
+ 		es->printNodes = false;
+ 		es->printAnalyze = true;
+ 		es->pstmt = queryDesc->plannedstmt;
+ 		es->rtable = queryDesc->plannedstmt->rtable;
+ 
+ 		initStringInfo(&buf);
+ 		explain_outNode(&buf,
+ 						queryDesc->plannedstmt->planTree,
+ 						queryDesc->planstate,
+ 						NULL, 0, es);
+ 
+ 		appendStringInfo(&buf, "Query runtime: %.3f ms",
+ 						 1000.0 * totaltime);
+ 
+ 		ereport(DEBUG1,
+ 				(errmsg("------------------- query plan -------------------"),
+ 				 errdetail("%s", buf.data)));
+ 
+ 		pfree(buf.data);
+ 		pfree(es);
+ 	}
  
  	return result;
  }
*** ./src/backend/utils/misc/guc.c.orig	Sun Jan 27 19:12:28 2008
--- ./src/backend/utils/misc/guc.c	Sun Jan 27 21:52:58 2008
***************
*** 193,198 ****
--- 193,199 ----
  bool		Debug_print_rewritten = false;
  bool		Debug_pretty_print = false;
  bool		Explain_pretty_print = true;
+ bool		Debug_explain_plan = false;
  
  bool		log_parser_stats = false;
  bool		log_planner_stats = false;
***************
*** 682,687 ****
--- 683,696 ----
  			NULL
  		},
  		&Debug_pretty_print,
+ 		false, NULL, NULL
+ 	},
+ 	{
+ 		{"debug_explain_plan", PGC_USERSET, LOGGING_WHAT,
+ 			gettext_noop("Explains the execution plan to server log."),
+ 			NULL
+ 		},
+ 		&Debug_explain_plan,
  		false, NULL, NULL
  	},
  	{
*** ./src/backend/utils/misc/postgresql.conf.sample.orig	Sun Jan 27 19:12:28 2008
--- ./src/backend/utils/misc/postgresql.conf.sample	Sun Jan 27 21:54:26 2008
***************
*** 323,328 ****
--- 323,329 ----
  #debug_print_rewritten = off
  #debug_print_plan = off
  #debug_pretty_print = off
+ #debug_explain_plan = off
  #log_checkpoints = off
  #log_connections = off
  #log_disconnections = off
*** ./src/include/commands/explain.h.orig	Tue Jan  1 19:45:58 2008
--- ./src/include/commands/explain.h	Sat Jan 26 11:53:48 2008
***************
*** 14,19 ****
--- 14,30 ----
  #define EXPLAIN_H
  
  #include "executor/executor.h"
+ #include "executor/instrument.h"
+ 
+ typedef struct ExplainState
+ {
+ 	/* options */
+ 	bool		printNodes;		/* do nodeToString() too */
+ 	bool		printAnalyze;	/* print actual times */
+ 	/* other states */
+ 	PlannedStmt *pstmt;			/* top of plan */
+ 	List	   *rtable;			/* range table */
+ } ExplainState;
  
  /* Hook for plugins to get control in ExplainOneQuery() */
  typedef void (*ExplainOneQuery_hook_type) (Query *query,
***************
*** 40,44 ****
--- 51,62 ----
  
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
+ 
+ extern void explain_outNode(StringInfo str,
+ 				Plan *plan, PlanState *planstate,
+ 				Plan *outer_plan,
+ 				int indent, ExplainState *es);
+ 
+ extern double elapsed_time(instr_time *starttime);
  
  #endif   /* EXPLAIN_H */
*** ./src/include/utils/guc.h.orig	Tue Jan  1 19:46:00 2008
--- ./src/include/utils/guc.h	Sat Jan 26 11:51:42 2008
***************
*** 117,122 ****
--- 117,123 ----
  extern bool Debug_print_rewritten;
  extern bool Debug_pretty_print;
  extern bool Explain_pretty_print;
+ extern bool Debug_explain_plan;
  
  extern bool log_parser_stats;
  extern bool log_planner_stats;
#2Marko Kreen
markokr@gmail.com
In reply to: Dean Rasheed (#1)

On 6/30/08, Dean Rasheed <dean_rasheed@hotmail.com> wrote:

This is a small patch I posted a few months back, and then kinda forgot
about / got distracted with other things.

Is there any interest in this? If so I am willing to put more work into
it, if people like it or have suggested improvements. Otherwise I'll let it
drop.

+1 for including it in whatever form. It is useful.

We actually already use in live settings (8.2 / 8,3).

(The last example shows foo() being called during the planning of this
query, which explains why it is called twice during the previous execution)

Simon Riggs reviewed this last time and said that what this patch
currently does is probably not exactly what is wanted for PostgreSQL.
Possible improvements might be to integrate this with the EXPLAIN
command (eg. EXPLAIN TRACE query) and have a separate parameter
(log_explain) for logging purposes.

I don't have strong opinion either way, It seems its more question
on style than any technical details. Just that plain EXPLAN MORE is
not enough, it would bo good have a way to turn it on in global/session
level for all queries.

--
marko

#3Alex Hunsaker
badalex@gmail.com
In reply to: Dean Rasheed (#1)

On Mon, Jun 30, 2008 at 6:34 AM, Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Hi,

This is a small patch I posted a few months back, and then kinda forgot
about / got distracted with other things.

Is there any interest in this? If so I am willing to put more work into
it, if people like it or have suggested improvements. Otherwise I'll let it
drop.

Here's what is does:

As it stands, basically it's just another debug parameter, called
debug_explain_plan, similar to debug_print_plan, except that the
output is in the form of EXPLAIN ANALYSE.

<snip>

Simon Riggs reviewed this last time and said that what this patch
currently does is probably not exactly what is wanted for PostgreSQL.
Possible improvements might be to integrate this with the EXPLAIN
command (eg. EXPLAIN TRACE query) and have a separate parameter
(log_explain) for logging purposes.

Comments?

Its certainly not useful to *me* in its current form. It would
produce way to much (usless) output. However if it were tied to
log_min_duration_statement so I get auto explains for long running
queries... That would be very useful indeed. Even if it has to
explain everything just to toss out the explain if it did not meet
log_min_duration_statement. Unless I missed something and thats
exactly what it does?

#4Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Alex Hunsaker (#3)

Its certainly not useful to *me* in its current form. It would
produce way to much (usless) output. However if it were tied to
log_min_duration_statement so I get auto explains for long running
queries... That would be very useful indeed. Even if it has to
explain everything just to toss out the explain if it did not meet
log_min_duration_statement. Unless I missed something and thats
exactly what it does?

Thanks for the feedback. I agree, it does need a way to limit the
output, and target just the slow-running queries.

I also remember now the problem I had last time:- since this debug
output is produced at a lower level than the other statement logging
(so it can explain *all* SQL executed, not just top-level statements), it
is difficult to control using the normal statement logging parameters.

It would be easy to add another parameter, debug_explain_min_duration,
specific to this option, to limit it to slow low-level queries.

This would allow setting debug_explain_min_duration to be smaller than
log_min_duration_statement, which makes sense, since the latter
controls logging of top-level statements which may result in multiple
low-level queries.

Doing it this way would mean instrumenting all queries, but only
explaining the slow ones, when debug_explain_plan is on.
I'll have a play and see how it goes...

Regards, Dean

_________________________________________________________________
Live Search Charades - guess correctly and find hidden videos
http://www.searchcharades.com/

#5Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Dean Rasheed (#4)
1 attachment(s)

Here is an updated version of the patch, with a debug_explain_min_duration
parameter to allow explaining of just slow-running queries. I've also incorporated
a couple of Simon Riggs' suggestions for formatting the output better.

Do I need to post this to -patches, or is that now obsolete?

Regards, Dean

----------------------------------------

From: dean_rasheed@hotmail.com
To: badalex@gmail.com
CC: pgsql-hackers@postgresql.org
Subject: RE: [HACKERS] Auto-explain patch
Date: Wed, 2 Jul 2008 19:42:06 +0000

Its certainly not useful to *me* in its current form. It would
produce way to much (usless) output. However if it were tied to
log_min_duration_statement so I get auto explains for long running
queries... That would be very useful indeed. Even if it has to
explain everything just to toss out the explain if it did not meet
log_min_duration_statement. Unless I missed something and thats
exactly what it does?

Thanks for the feedback. I agree, it does need a way to limit the
output, and target just the slow-running queries.

I also remember now the problem I had last time:- since this debug
output is produced at a lower level than the other statement logging
(so it can explain *all* SQL executed, not just top-level statements), it
is difficult to control using the normal statement logging parameters.

It would be easy to add another parameter, debug_explain_min_duration,
specific to this option, to limit it to slow low-level queries.

This would allow setting debug_explain_min_duration to be smaller than
log_min_duration_statement, which makes sense, since the latter
controls logging of top-level statements which may result in multiple
low-level queries.

Doing it this way would mean instrumenting all queries, but only
explaining the slow ones, when debug_explain_plan is on.
I'll have a play and see how it goes...

Regards, Dean

_________________________________________________________________
Live Search Charades - guess correctly and find hidden videos
http://www.searchcharades.com/

_________________________________________________________________
The next generation of Windows Live is here
http://www.windowslive.co.uk/get-live

Attachments:

auto-explain2.patchtext/x-patchDownload
*** ./doc/src/sgml/config.sgml.orig	2008-03-11 16:59:09.000000000 +0000
--- ./doc/src/sgml/config.sgml	2008-07-03 14:20:15.000000000 +0100
***************
*** 2674,2679 ****
--- 2674,2700 ----
         </listitem>
        </varlistentry>
  
+      <varlistentry id="guc-debug-explain-min-duration" xreflabel="debug_explain_min_duration">
+       <term><varname>debug_explain_min_duration</varname> (<type>integer</type>)</term>
+       <indexterm>
+        <primary><varname>debug_explain_min_duration</> configuration parameter</primary>
+       </indexterm>
+       <listitem>
+        <para>
+         This option, together with <xref linkend="guc-debug-explain-plan">,
+         enables logging of debug messages explaining all SQL queries which
+         run for at least the specified number of milliseconds. Setting this
+         to zero (the default) will cause all statement execution plans to be
+         explained, when <xref linkend="guc-debug-explain-plan"> is on.
+        </para>
+ 
+        <para>
+         When <xref linkend="guc-debug-explain-plan"> if off, no statements
+         are explained, and this parameter has no effect.
+        </para>
+       </listitem>
+      </varlistentry>
+ 
       <varlistentry id="guc-silent-mode" xreflabel="silent_mode">
        <term><varname>silent_mode</varname> (<type>boolean</type>)</term>
        <indexterm>
***************
*** 2794,2799 ****
--- 2815,2822 ----
        <term><varname>debug_print_rewritten</varname> (<type>boolean</type>)</term>
        <term><varname>debug_print_plan</varname> (<type>boolean</type>)</term>
        <term><varname>debug_pretty_print</varname> (<type>boolean</type>)</term>
+       <term><anchor id="guc-debug-explain-plan" xreflabel="debug_explain_plan">
+        <varname>debug_explain_plan</varname> (<type>boolean</type>)</term>
        <indexterm>
         <primary><varname>debug_print_parse</> configuration parameter</primary>
        </indexterm>
***************
*** 2806,2811 ****
--- 2829,2837 ----
        <indexterm>
         <primary><varname>debug_pretty_print</> configuration parameter</primary>
        </indexterm>
+       <indexterm>
+        <primary><varname>debug_explain_plan</> configuration parameter</primary>
+       </indexterm>
        <listitem>
         <para>
          These parameters enable various debugging output to be emitted.
***************
*** 2813,2824 ****
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
        </listitem>
       </varlistentry>
  
--- 2839,2867 ----
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>debug_explain_plan</varname> prints
!         the plan for each executed query in the same format as
!         <command>EXPLAIN ANALYZE</>. This includes queries executed from
!         within functions.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
+ 
+        <note>
+         <para>
+          The reports produced by <varname>debug_explain_plan</varname>
+          are produced at a lower level in the database, as each query
+          is executed, including queries executed from functions, so
+          the output may be more verbose that of <command>EXPLAIN ANALYZE</>
+          and the timings may differ. When this option is used together
+          with <xref linkend="guc-debug-explain-min-duration">, all queries
+          will be instrumented, but only those which run for at least the
+          specified number of milliseconds will have their execution plans
+          reported. 
+         </para>
+        </note>
        </listitem>
       </varlistentry>
  
*** ./src/backend/commands/explain.c.orig	2008-01-01 19:45:49.000000000 +0000
--- ./src/backend/commands/explain.c	2008-06-27 12:06:19.000000000 +0100
***************
*** 39,65 ****
  /* Hook for plugins to get control in explain_get_index_name() */
  explain_get_index_name_hook_type explain_get_index_name_hook = NULL;
  
- 
- typedef struct ExplainState
- {
- 	/* options */
- 	bool		printNodes;		/* do nodeToString() too */
- 	bool		printAnalyze;	/* print actual times */
- 	/* other states */
- 	PlannedStmt *pstmt;			/* top of plan */
- 	List	   *rtable;			/* range table */
- } ExplainState;
- 
  static void ExplainOneQuery(Query *query, ExplainStmt *stmt,
  				const char *queryString,
  				ParamListInfo params, TupOutputState *tstate);
  static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
  				StringInfo buf);
- static double elapsed_time(instr_time *starttime);
- static void explain_outNode(StringInfo str,
- 				Plan *plan, PlanState *planstate,
- 				Plan *outer_plan,
- 				int indent, ExplainState *es);
  static void show_scan_qual(List *qual, const char *qlabel,
  			   int scanrelid, Plan *outer_plan, Plan *inner_plan,
  			   StringInfo str, int indent, ExplainState *es);
--- 39,49 ----
***************
*** 402,408 ****
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! static double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
--- 386,392 ----
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
***************
*** 436,442 ****
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! static void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
--- 420,426 ----
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
*** ./src/backend/executor/execMain.c.orig	2008-04-21 04:49:51.000000000 +0100
--- ./src/backend/executor/execMain.c	2008-07-03 09:49:52.000000000 +0100
***************
*** 39,44 ****
--- 39,45 ----
  #include "catalog/heap.h"
  #include "catalog/namespace.h"
  #include "catalog/toasting.h"
+ #include "commands/explain.h"
  #include "commands/tablespace.h"
  #include "commands/trigger.h"
  #include "executor/execdebug.h"
***************
*** 52,57 ****
--- 53,59 ----
  #include "utils/acl.h"
  #include "utils/lsyscache.h"
  #include "utils/memutils.h"
+ #include "utils/guc.h"
  
  
  typedef struct evalPlanQual
***************
*** 185,190 ****
--- 187,200 ----
  	}
  
  	/*
+ 	 * If we are explaining all queries, enable instrumentation.
+ 	 * Even if we are only explaining the slow queries, we still
+ 	 * need to instrument them all.
+ 	 */
+ 	if (Debug_explain_plan)
+ 		queryDesc->doInstrument = true;
+ 
+ 	/*
  	 * Copy other important information into the EState
  	 */
  	estate->es_snapshot = queryDesc->snapshot;
***************
*** 227,232 ****
--- 237,245 ----
  	bool		sendTuples;
  	TupleTableSlot *result;
  	MemoryContext oldcontext;
+ 	bool		explainPlan = Debug_explain_plan;
+ 	instr_time	starttime;
+ 	double		totaltime;
  
  	/* sanity checks */
  	Assert(queryDesc != NULL);
***************
*** 236,241 ****
--- 249,265 ----
  	Assert(estate != NULL);
  
  	/*
+ 	 * If explaining all queries, record the start time before running
+ 	 * the query. This is not quite the same as EXPLAIN ANALYSE, which
+ 	 * starts timing before ExecutorStart(). Here we are only timing
+ 	 * how long the query takes to run, once initialised and also
+ 	 * excluding any triggers (which may themselves run queries which
+ 	 * will be instrumented separately).
+ 	 */
+ 	if (explainPlan)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/*
  	 * Switch into per-query memory context
  	 */
  	oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
***************
*** 279,284 ****
--- 303,341 ----
  
  	MemoryContextSwitchTo(oldcontext);
  
+ 	/*
+ 	 * If explaining queries and this one was slow enough, log the
+ 	 * execution plan using the same format as EXPLAIN ANALYSE
+ 	 */
+ 	totaltime = elapsed_time(&starttime);
+ 	if (explainPlan && totaltime * 1000 >= Debug_explain_min_duration)
+ 	{
+ 		ExplainState *es;
+ 		StringInfoData buf;
+ 
+ 		es = (ExplainState *) palloc0(sizeof(ExplainState));
+ 
+ 		es->printNodes = false;
+ 		es->printAnalyze = true;
+ 		es->pstmt = queryDesc->plannedstmt;
+ 		es->rtable = queryDesc->plannedstmt->rtable;
+ 
+ 		initStringInfo(&buf);
+ 		explain_outNode(&buf,
+ 						queryDesc->plannedstmt->planTree,
+ 						queryDesc->planstate,
+ 						NULL, 0, es);
+ 
+ 		appendStringInfo(&buf, "Query runtime: %.3f ms",
+ 						 1000.0 * totaltime);
+ 
+ 		ereport(DEBUG1,
+ 				(errmsg("Query plan:\n%s", buf.data)));
+ 
+ 		pfree(buf.data);
+ 		pfree(es);
+ 	}
+ 
  	return result;
  }
  
*** ./src/backend/utils/misc/guc.c.orig	2008-05-26 19:54:36.000000000 +0100
--- ./src/backend/utils/misc/guc.c	2008-07-03 09:49:30.000000000 +0100
***************
*** 194,199 ****
--- 194,201 ----
  bool		Debug_print_rewritten = false;
  bool		Debug_pretty_print = false;
  bool		Explain_pretty_print = true;
+ bool		Debug_explain_plan = false;
+ int			Debug_explain_min_duration = 0;
  
  bool		log_parser_stats = false;
  bool		log_planner_stats = false;
***************
*** 686,691 ****
--- 688,701 ----
  		false, NULL, NULL
  	},
  	{
+ 		{"debug_explain_plan", PGC_USERSET, LOGGING_WHAT,
+ 			gettext_noop("Explains the execution plan to server log."),
+ 			NULL
+ 		},
+ 		&Debug_explain_plan,
+ 		false, NULL, NULL
+ 	},
+ 	{
  		{"log_parser_stats", PGC_SUSET, STATS_MONITORING,
  			gettext_noop("Writes parser performance statistics to the server log."),
  			NULL
***************
*** 1567,1572 ****
--- 1577,1593 ----
  	},
  
  	{
+ 		{"debug_explain_min_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum execution time above which "
+ 						 "statements will be explained."),
+ 			gettext_noop("0 causes all statements to be explained."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&Debug_explain_min_duration,
+ 		0, 0, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
  		{"log_autovacuum_min_duration", PGC_SIGHUP, LOGGING_WHAT,
  			gettext_noop("Sets the minimum execution time above which "
  						 "autovacuum actions will be logged."),
*** ./src/backend/utils/misc/postgresql.conf.sample.orig	2008-01-30 18:35:55.000000000 +0000
--- ./src/backend/utils/misc/postgresql.conf.sample	2008-07-03 10:08:57.000000000 +0100
***************
*** 313,318 ****
--- 313,322 ----
  					# and their durations, > 0 logs only
  					# statements running at least this time.
  
+ #debug_explain_min_duration = 0		# limits the output of debug_explain_plan
+ 					# to queries which run for at least the
+ 					# the specified number of milliseconds.
+ 
  #silent_mode = off			# DO NOT USE without syslog or
  					# logging_collector
  					# (change requires restart)
***************
*** 323,328 ****
--- 327,333 ----
  #debug_print_rewritten = off
  #debug_print_plan = off
  #debug_pretty_print = off
+ #debug_explain_plan = off
  #log_checkpoints = off
  #log_connections = off
  #log_disconnections = off
*** ./src/include/commands/explain.h.orig	2008-01-01 19:45:57.000000000 +0000
--- ./src/include/commands/explain.h	2008-06-27 12:06:19.000000000 +0100
***************
*** 14,19 ****
--- 14,30 ----
  #define EXPLAIN_H
  
  #include "executor/executor.h"
+ #include "executor/instrument.h"
+ 
+ typedef struct ExplainState
+ {
+ 	/* options */
+ 	bool		printNodes;		/* do nodeToString() too */
+ 	bool		printAnalyze;	/* print actual times */
+ 	/* other states */
+ 	PlannedStmt *pstmt;			/* top of plan */
+ 	List	   *rtable;			/* range table */
+ } ExplainState;
  
  /* Hook for plugins to get control in ExplainOneQuery() */
  typedef void (*ExplainOneQuery_hook_type) (Query *query,
***************
*** 41,44 ****
--- 52,62 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void explain_outNode(StringInfo str,
+ 				Plan *plan, PlanState *planstate,
+ 				Plan *outer_plan,
+ 				int indent, ExplainState *es);
+ 
+ extern double elapsed_time(instr_time *starttime);
+ 
  #endif   /* EXPLAIN_H */
*** ./src/include/utils/guc.h.orig	2008-01-01 19:45:59.000000000 +0000
--- ./src/include/utils/guc.h	2008-07-02 20:46:55.000000000 +0100
***************
*** 117,122 ****
--- 117,124 ----
  extern bool Debug_print_rewritten;
  extern bool Debug_pretty_print;
  extern bool Explain_pretty_print;
+ extern bool Debug_explain_plan;
+ extern int  Debug_explain_min_duration;
  
  extern bool log_parser_stats;
  extern bool log_planner_stats;
#6Alex Hunsaker
badalex@gmail.com
In reply to: Dean Rasheed (#5)

On Thu, Jul 3, 2008 at 10:58 AM, Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Here is an updated version of the patch, with a debug_explain_min_duration
parameter to allow explaining of just slow-running queries. I've also incorporated
a couple of Simon Riggs' suggestions for formatting the output better.

Do I need to post this to -patches, or is that now obsolete?

Yes its obsolete now, instead add it to July's commit fest at
http://wiki.postgresql.org/wiki/CommitFest:July.

#7Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#5)

On Thu, 2008-07-03 at 16:58 +0000, Dean Rasheed wrote:

Here is an updated version of the patch

Dean,

I'm getting 4 chunks fail when trying to apply your patch onto CVS HEAD.

I'm sure its just a little bitrot. Can you update the patch please?

Thanks,

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#8Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Simon Riggs (#7)
1 attachment(s)

OK, this one should work against CVS HEAD.

Dean.

Subject: Re: [HACKERS] Auto-explain patch> From: simon@2ndquadrant.com> To: dean_rasheed@hotmail.com> CC: pgsql-hackers@postgresql.org> Date: Sun, 6 Jul 2008 16:42:55 +0100> > > On Thu, 2008-07-03 at 16:58 +0000, Dean Rasheed wrote:> > Here is an updated version of the patch> > Dean,> > I'm getting 4 chunks fail when trying to apply your patch onto CVS HEAD.> > I'm sure its just a little bitrot. Can you update the patch please? > > Thanks,> > -- > Simon Riggs www.2ndQuadrant.com> PostgreSQL Training, Services and Support>

_________________________________________________________________
The John Lewis Clearance - save up to 50% with FREE delivery
http://clk.atdmt.com/UKM/go/101719806/direct/01/

Attachments:

auto-explain2.patchtext/plainDownload
*** ./doc/src/sgml/config.sgml.orig	2008-03-11 16:59:09.000000000 +0000
--- ./doc/src/sgml/config.sgml	2008-07-03 14:20:15.000000000 +0100
***************
*** 2674,2679 ****
--- 2674,2700 ----
         </listitem>
        </varlistentry>
  
+      <varlistentry id="guc-debug-explain-min-duration" xreflabel="debug_explain_min_duration">
+       <term><varname>debug_explain_min_duration</varname> (<type>integer</type>)</term>
+       <indexterm>
+        <primary><varname>debug_explain_min_duration</> configuration parameter</primary>
+       </indexterm>
+       <listitem>
+        <para>
+         This option, together with <xref linkend="guc-debug-explain-plan">,
+         enables logging of debug messages explaining all SQL queries which
+         run for at least the specified number of milliseconds. Setting this
+         to zero (the default) will cause all statement execution plans to be
+         explained, when <xref linkend="guc-debug-explain-plan"> is on.
+        </para>
+ 
+        <para>
+         When <xref linkend="guc-debug-explain-plan"> if off, no statements
+         are explained, and this parameter has no effect.
+        </para>
+       </listitem>
+      </varlistentry>
+ 
       <varlistentry id="guc-silent-mode" xreflabel="silent_mode">
        <term><varname>silent_mode</varname> (<type>boolean</type>)</term>
        <indexterm>
***************
*** 2794,2799 ****
--- 2815,2822 ----
        <term><varname>debug_print_rewritten</varname> (<type>boolean</type>)</term>
        <term><varname>debug_print_plan</varname> (<type>boolean</type>)</term>
        <term><varname>debug_pretty_print</varname> (<type>boolean</type>)</term>
+       <term><anchor id="guc-debug-explain-plan" xreflabel="debug_explain_plan">
+        <varname>debug_explain_plan</varname> (<type>boolean</type>)</term>
        <indexterm>
         <primary><varname>debug_print_parse</> configuration parameter</primary>
        </indexterm>
***************
*** 2806,2811 ****
--- 2829,2837 ----
        <indexterm>
         <primary><varname>debug_pretty_print</> configuration parameter</primary>
        </indexterm>
+       <indexterm>
+        <primary><varname>debug_explain_plan</> configuration parameter</primary>
+       </indexterm>
        <listitem>
         <para>
          These parameters enable various debugging output to be emitted.
***************
*** 2813,2824 ****
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
        </listitem>
       </varlistentry>
  
--- 2839,2867 ----
          the resulting parse tree, the query rewriter output, or the
          execution plan.  <varname>debug_pretty_print</varname> indents
          these displays to produce a more readable but much longer
!         output format.  <varname>debug_explain_plan</varname> prints
!         the plan for each executed query in the same format as
!         <command>EXPLAIN ANALYZE</>. This includes queries executed from
!         within functions.  <varname>client_min_messages</varname> or
          <varname>log_min_messages</varname> must be
          <literal>DEBUG1</literal> or lower to actually send this output
          to the client or the server log, respectively.
          These parameters are off by default.
         </para>
+ 
+        <note>
+         <para>
+          The reports produced by <varname>debug_explain_plan</varname>
+          are produced at a lower level in the database, as each query
+          is executed, including queries executed from functions, so
+          the output may be more verbose that of <command>EXPLAIN ANALYZE</>
+          and the timings may differ. When this option is used together
+          with <xref linkend="guc-debug-explain-min-duration">, all queries
+          will be instrumented, but only those which run for at least the
+          specified number of milliseconds will have their execution plans
+          reported. 
+         </para>
+        </note>
        </listitem>
       </varlistentry>
  
*** ./src/backend/commands/explain.c.orig	2008-01-01 19:45:49.000000000 +0000
--- ./src/backend/commands/explain.c	2008-06-27 12:06:19.000000000 +0100
***************
*** 39,65 ****
  /* Hook for plugins to get control in explain_get_index_name() */
  explain_get_index_name_hook_type explain_get_index_name_hook = NULL;
  
- 
- typedef struct ExplainState
- {
- 	/* options */
- 	bool		printTList;		/* print plan targetlists */
- 	bool		printAnalyze;	/* print actual times */
- 	/* other states */
- 	PlannedStmt *pstmt;			/* top of plan */
- 	List	   *rtable;			/* range table */
- } ExplainState;
- 
  static void ExplainOneQuery(Query *query, ExplainStmt *stmt,
  				const char *queryString,
  				ParamListInfo params, TupOutputState *tstate);
  static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
  				StringInfo buf);
- static double elapsed_time(instr_time *starttime);
- static void explain_outNode(StringInfo str,
- 				Plan *plan, PlanState *planstate,
- 				Plan *outer_plan,
- 				int indent, ExplainState *es);
  static void show_plan_tlist(Plan *plan,
  							StringInfo str, int indent, ExplainState *es);
  static void show_scan_qual(List *qual, const char *qlabel,
--- 39,49 ----
***************
*** 402,408 ****
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! static double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
--- 386,392 ----
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
***************
*** 436,442 ****
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! static void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
--- 420,426 ----
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
*** ./src/backend/executor/execMain.c.orig	2008-04-21 04:49:51.000000000 +0100
--- ./src/backend/executor/execMain.c	2008-07-03 09:49:52.000000000 +0100
***************
*** 39,44 ****
--- 39,45 ----
  #include "catalog/heap.h"
  #include "catalog/namespace.h"
  #include "catalog/toasting.h"
+ #include "commands/explain.h"
  #include "commands/tablespace.h"
  #include "commands/trigger.h"
  #include "executor/execdebug.h"
***************
*** 52,57 ****
--- 53,59 ----
  #include "utils/acl.h"
  #include "utils/lsyscache.h"
  #include "utils/memutils.h"
+ #include "utils/guc.h"
  #include "utils/snapmgr.h"
  #include "utils/tqual.h"
  
***************
*** 185,190 ****
--- 187,200 ----
  	}
  
  	/*
+ 	 * If we are explaining all queries, enable instrumentation.
+ 	 * Even if we are only explaining the slow queries, we still
+ 	 * need to instrument them all.
+ 	 */
+ 	if (Debug_explain_plan)
+ 		queryDesc->doInstrument = true;
+ 
+ 	/*
  	 * Copy other important information into the EState
  	 */
  	estate->es_snapshot = queryDesc->snapshot;
***************
*** 227,232 ****
--- 237,245 ----
  	bool		sendTuples;
  	TupleTableSlot *result;
  	MemoryContext oldcontext;
+ 	bool		explainPlan = Debug_explain_plan;
+ 	instr_time	starttime;
+ 	double		totaltime;
  
  	/* sanity checks */
  	Assert(queryDesc != NULL);
***************
*** 236,241 ****
--- 249,265 ----
  	Assert(estate != NULL);
  
  	/*
+ 	 * If explaining all queries, record the start time before running
+ 	 * the query. This is not quite the same as EXPLAIN ANALYSE, which
+ 	 * starts timing before ExecutorStart(). Here we are only timing
+ 	 * how long the query takes to run, once initialised and also
+ 	 * excluding any triggers (which may themselves run queries which
+ 	 * will be instrumented separately).
+ 	 */
+ 	if (explainPlan)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/*
  	 * Switch into per-query memory context
  	 */
  	oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
***************
*** 279,284 ****
--- 303,341 ----
  
  	MemoryContextSwitchTo(oldcontext);
  
+ 	/*
+ 	 * If explaining queries and this one was slow enough, log the
+ 	 * execution plan using the same format as EXPLAIN ANALYSE
+ 	 */
+ 	totaltime = elapsed_time(&starttime);
+ 	if (explainPlan && totaltime * 1000 >= Debug_explain_min_duration)
+ 	{
+ 		ExplainState *es;
+ 		StringInfoData buf;
+ 
+ 		es = (ExplainState *) palloc0(sizeof(ExplainState));
+ 
+ 		es->printTList = false;
+ 		es->printAnalyze = true;
+ 		es->pstmt = queryDesc->plannedstmt;
+ 		es->rtable = queryDesc->plannedstmt->rtable;
+ 
+ 		initStringInfo(&buf);
+ 		explain_outNode(&buf,
+ 						queryDesc->plannedstmt->planTree,
+ 						queryDesc->planstate,
+ 						NULL, 0, es);
+ 
+ 		appendStringInfo(&buf, "Query runtime: %.3f ms",
+ 						 1000.0 * totaltime);
+ 
+ 		ereport(DEBUG1,
+ 				(errmsg("Query plan:\n%s", buf.data)));
+ 
+ 		pfree(buf.data);
+ 		pfree(es);
+ 	}
+ 
  	return result;
  }
  
*** ./src/backend/utils/misc/guc.c.orig	2008-05-26 19:54:36.000000000 +0100
--- ./src/backend/utils/misc/guc.c	2008-07-03 09:49:30.000000000 +0100
***************
*** 194,199 ****
--- 194,201 ----
  bool		Debug_print_parse = false;
  bool		Debug_print_rewritten = false;
  bool		Debug_pretty_print = false;
+ bool		Debug_explain_plan = false;
+ int			Debug_explain_min_duration = 0;
  
  bool		log_parser_stats = false;
  bool		log_planner_stats = false;
***************
*** 686,691 ****
--- 688,701 ----
  		false, NULL, NULL
  	},
  	{
+ 		{"debug_explain_plan", PGC_USERSET, LOGGING_WHAT,
+ 			gettext_noop("Explains the execution plan to server log."),
+ 			NULL
+ 		},
+ 		&Debug_explain_plan,
+ 		false, NULL, NULL
+ 	},
+ 	{
  		{"log_parser_stats", PGC_SUSET, STATS_MONITORING,
  			gettext_noop("Writes parser performance statistics to the server log."),
  			NULL
***************
*** 1567,1572 ****
--- 1577,1593 ----
  	},
  
  	{
+ 		{"debug_explain_min_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum execution time above which "
+ 						 "statements will be explained."),
+ 			gettext_noop("0 causes all statements to be explained."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&Debug_explain_min_duration,
+ 		0, 0, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
  		{"log_autovacuum_min_duration", PGC_SIGHUP, LOGGING_WHAT,
  			gettext_noop("Sets the minimum execution time above which "
  						 "autovacuum actions will be logged."),
*** ./src/backend/utils/misc/postgresql.conf.sample.orig	2008-01-30 18:35:55.000000000 +0000
--- ./src/backend/utils/misc/postgresql.conf.sample	2008-07-03 10:08:57.000000000 +0100
***************
*** 313,318 ****
--- 313,322 ----
  					# and their durations, > 0 logs only
  					# statements running at least this time.
  
+ #debug_explain_min_duration = 0		# limits the output of debug_explain_plan
+ 					# to queries which run for at least the
+ 					# the specified number of milliseconds.
+ 
  #silent_mode = off			# DO NOT USE without syslog or
  					# logging_collector
  					# (change requires restart)
***************
*** 323,328 ****
--- 327,333 ----
  #debug_print_rewritten = off
  #debug_print_plan = off
  #debug_pretty_print = off
+ #debug_explain_plan = off
  #log_checkpoints = off
  #log_connections = off
  #log_disconnections = off
*** ./src/include/commands/explain.h.orig	2008-01-01 19:45:57.000000000 +0000
--- ./src/include/commands/explain.h	2008-06-27 12:06:19.000000000 +0100
***************
*** 14,19 ****
--- 14,30 ----
  #define EXPLAIN_H
  
  #include "executor/executor.h"
+ #include "executor/instrument.h"
+ 
+ typedef struct ExplainState
+ {
+ 	/* options */
+ 	bool		printTList;		/* print plan targetlists */
+ 	bool		printAnalyze;	/* print actual times */
+ 	/* other states */
+ 	PlannedStmt *pstmt;			/* top of plan */
+ 	List	   *rtable;			/* range table */
+ } ExplainState;
  
  /* Hook for plugins to get control in ExplainOneQuery() */
  typedef void (*ExplainOneQuery_hook_type) (Query *query,
***************
*** 41,44 ****
--- 52,62 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void explain_outNode(StringInfo str,
+ 				Plan *plan, PlanState *planstate,
+ 				Plan *outer_plan,
+ 				int indent, ExplainState *es);
+ 
+ extern double elapsed_time(instr_time *starttime);
+ 
  #endif   /* EXPLAIN_H */
*** ./src/include/utils/guc.h.orig	2008-01-01 19:45:59.000000000 +0000
--- ./src/include/utils/guc.h	2008-07-02 20:46:55.000000000 +0100
***************
*** 117,122 ****
--- 117,124 ----
  extern bool Debug_print_parse;
  extern bool Debug_print_rewritten;
  extern bool Debug_pretty_print;
+ extern bool Debug_explain_plan;
+ extern int  Debug_explain_min_duration;
  
  extern bool log_parser_stats;
  extern bool log_planner_stats;
#9Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#8)

On Sun, 2008-07-06 at 17:58 +0000, Dean Rasheed wrote:

OK, this one should work against CVS HEAD.

OK, still getting some offsets, but it applies.

The patch now outputs things in a useful way to avoid a flood of
information, that's good.

The code seems fine, but it doesn't link up with the other logging code
in postgres.c. That split also seems to drive the way the parameters
work, so you have another set of parameters mirroring the
log_min_duration_statement infrastructure. Fairly yuk, but I understand
why you've done this. It does however beg some serious questions about
the way we currently do logging.

The patch as it stands would produce EXPLAIN output for queries that
will never be logged, which isn't very useful. So Dean, right now I feel
this version of the patch must be rejected while we discuss how things
might work differently. Not an issue with the patch, so much as issues
with the code being patched.

So thinking about the issues we can either ignore them or tackle them:

Ignoring problem option: Rewrite this patch as an executor plugin, using
Itagaki Takahiro's new infrastructure. It's an option, but I guess Dave
will have something to say about that...

Fix problems: Having logging in two separate places is ugly and leads to
the problem of logging EXPLAINs and SQL completely separately, which
isn't very useful.

If we put all the logging code in the executor we miss out any parsing
and planning time but we can log statements and explains for SPI queries
also.

We could move the logging of the EXPLAIN to postgres.c, but then we'd
need to make querydescs live a little longer than desirable, or we
trigger the EXPLAIN code using some guess at what the planning overhead
was, a horrible kluge.

Neither way is clean. So how do we do it?

My proposal would be to introduce some new logging statements that work
similarly to, yet as an alternative to log_min_duration_statement.

* log_min_planner_duration - logs any statement whose planning time
exceeds the value in ms (-1 is off).
e.g. LOG planner time: %s ms statement: %s

* log_min_executor_duration - logs any execution whose execution time
exceeds the value in ms (-1 is off). We need to pass query text through
to the executor, depending upon whether its a portal's source text or an
incoming simple querystring. (This is similar, but not quite the same as
current patch's debug_explain_min_duration).
e.g. LOG executor time: %s ms statement: %s

* log_explain = on | off - logs EXPLAIN if either
log_min_executor_duration or log_min_planner_duration triggers. We
output the explain on the same line as the SQL statement, so we don't
need to play games to link up the output later. (This is similar to, but
not quite the same as current patch's debug_explain_plan).
e.g. LOG executor time: %s ms statement: %s

If both log_min_executor_duration and log_min_planner_duration trigger
then we get only one log line like this:
e.g. LOG planner time: %s ms executor time: %s ms statement: %s

We should call these parameters log_xxx not debug_xxx because we want
them for production logging.

If either of these new parameters is > -1 then
log_min_duration_statement does not trigger. (Or we simply make
log_min_duration_statement control the setting of the other two
parameters).

The log messages shown above allow us to differentiate between the
reasons the SQL statement has been logged, as well as providing the
details we want.

If we do this then:

* we can log EXPLAINs for any execution, plus we will always have the
SQL for any EXPLAIN

* we can log statements that take a long time to plan, yet a short time
to execute. That can be an overhead in some cases. It also neatly
sidelines the issue of where logging takes place. (We need to be able to
track planner time, if we think we might want to introduce options to
control planner effort, topic discussed on other recent thread, not
here)

* we can log SQL for any execution, even those via SPI. That means we'll
be able to log the SQL used in functions.

We'll probably need an option to log SQL from SPI or not. Things like RI
trigger code probably doesn't need to be logged, especially if the top
level statement is, like ALTER TABLE.

This alters current performance logging, so I expect discussion.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#10Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Simon Riggs (#9)

Simon, I like your proposal, and I think I can see how to code it
fairly easily.

There is one thing that it doesn't allow, however, which the debug_xxx
parameters do, and that is for a non-superuser to trace SQL used in
functions, from an interactive client session. For me, this is quite a
big thing, because I find it most convienient to turn these parameters
on while writing and tweaking stored procedures, and have the output
go straight to my psql window, without having to connect as a superuser
and trawl through log files.

So I suggest grouping these parameters in their own category
(eg. "sql_trace") and then having additional parameters to control
where the output would go. So the sql_trace parameters would be:

* sql_trace_min_planner_duration
* sql_trace_min_executor_duration
* sql_trace_explain_plan

and they would work exactly as you describe, except they would be
settable by normal users. Then the destination(s) for the statement
and EXPLAIN logging would be controlled by:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

* log_sql_trace = on | off - settable only by superusers. If this
parameter is on, the sql_trace will be logged as LOG output, and the
normal client sessions will see nothing (unless they set client_sql_trace).

If both client_sql_trace and log_sql_trace are on, then the output
goes to both locations, and if both are off, then the sql_trace
parameters would do nothing.

Dean

Subject: RE: [HACKERS] Auto-explain patch
From: simon@2ndquadrant.com
To: dean_rasheed@hotmail.com
CC: pgsql-hackers@postgresql.org
Date: Mon, 7 Jul 2008 18:03:15 +0100

On Sun, 2008-07-06 at 17:58 +0000, Dean Rasheed wrote:

OK, this one should work against CVS HEAD.

OK, still getting some offsets, but it applies.

The patch now outputs things in a useful way to avoid a flood of
information, that's good.

The code seems fine, but it doesn't link up with the other logging code
in postgres.c. That split also seems to drive the way the parameters
work, so you have another set of parameters mirroring the
log_min_duration_statement infrastructure. Fairly yuk, but I understand
why you've done this. It does however beg some serious questions about
the way we currently do logging.

The patch as it stands would produce EXPLAIN output for queries that
will never be logged, which isn't very useful. So Dean, right now I feel
this version of the patch must be rejected while we discuss how things
might work differently. Not an issue with the patch, so much as issues
with the code being patched.

So thinking about the issues we can either ignore them or tackle them:

Ignoring problem option: Rewrite this patch as an executor plugin, using
Itagaki Takahiro's new infrastructure. It's an option, but I guess Dave
will have something to say about that...

Fix problems: Having logging in two separate places is ugly and leads to
the problem of logging EXPLAINs and SQL completely separately, which
isn't very useful.

If we put all the logging code in the executor we miss out any parsing
and planning time but we can log statements and explains for SPI queries
also.

We could move the logging of the EXPLAIN to postgres.c, but then we'd
need to make querydescs live a little longer than desirable, or we
trigger the EXPLAIN code using some guess at what the planning overhead
was, a horrible kluge.

Neither way is clean. So how do we do it?

My proposal would be to introduce some new logging statements that work
similarly to, yet as an alternative to log_min_duration_statement.

* log_min_planner_duration - logs any statement whose planning time
exceeds the value in ms (-1 is off).
e.g. LOG planner time: %s ms statement: %s

* log_min_executor_duration - logs any execution whose execution time
exceeds the value in ms (-1 is off). We need to pass query text through
to the executor, depending upon whether its a portal's source text or an
incoming simple querystring. (This is similar, but not quite the same as
current patch's debug_explain_min_duration).
e.g. LOG executor time: %s ms statement: %s

* log_explain = on | off - logs EXPLAIN if either
log_min_executor_duration or log_min_planner_duration triggers. We
output the explain on the same line as the SQL statement, so we don't
need to play games to link up the output later. (This is similar to, but
not quite the same as current patch's debug_explain_plan).
e.g. LOG executor time: %s ms statement: %s

If both log_min_executor_duration and log_min_planner_duration trigger
then we get only one log line like this:
e.g. LOG planner time: %s ms executor time: %s ms statement: %s

We should call these parameters log_xxx not debug_xxx because we want
them for production logging.

If either of these new parameters is> -1 then
log_min_duration_statement does not trigger. (Or we simply make
log_min_duration_statement control the setting of the other two
parameters).

The log messages shown above allow us to differentiate between the
reasons the SQL statement has been logged, as well as providing the
details we want.

If we do this then:

* we can log EXPLAINs for any execution, plus we will always have the
SQL for any EXPLAIN

* we can log statements that take a long time to plan, yet a short time
to execute. That can be an overhead in some cases. It also neatly
sidelines the issue of where logging takes place. (We need to be able to
track planner time, if we think we might want to introduce options to
control planner effort, topic discussed on other recent thread, not
here)

* we can log SQL for any execution, even those via SPI. That means we'll
be able to log the SQL used in functions.

We'll probably need an option to log SQL from SPI or not. Things like RI
trigger code probably doesn't need to be logged, especially if the top
level statement is, like ALTER TABLE.

This alters current performance logging, so I expect discussion.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

_________________________________________________________________
Invite your Facebook friends to chat on Messenger
http://clk.atdmt.com/UKM/go/101719649/direct/01/

#11ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Dean Rasheed (#10)

Dean Rasheed <dean_rasheed@hotmail.com> wrote:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

In terms of security, is it ok to show normal users SQLs used in functions
that are owned by other users? Users can call not-owned functions only if
they have EXECUTE privilege on them. -- presently we can see function
bodies from pg_proc.prosrc freely, though.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#12Marko Kreen
markokr@gmail.com
In reply to: ITAGAKI Takahiro (#11)

On 7/9/08, ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:

Dean Rasheed <dean_rasheed@hotmail.com> wrote:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

In terms of security, is it ok to show normal users SQLs used in functions
that are owned by other users? Users can call not-owned functions only if
they have EXECUTE privilege on them. -- presently we can see function
bodies from pg_proc.prosrc freely, though.

Different owner is not a problem, but SECURITY DEFINER may be.

That can be solved by turning the setting off on entry to such function,
by non-superuser. Like we handle search_path.

--
marko

#13Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#10)

On Wed, 2008-07-09 at 09:11 +0000, Dean Rasheed wrote:

Simon, I like your proposal, and I think I can see how to code it
fairly easily.

There is one thing that it doesn't allow, however, which the debug_xxx
parameters do, and that is for a non-superuser to trace SQL used in
functions, from an interactive client session. For me, this is quite a
big thing, because I find it most convienient to turn these parameters
on while writing and tweaking stored procedures, and have the output
go straight to my psql window, without having to connect as a superuser
and trawl through log files.

Understood.

So I suggest grouping these parameters in their own category
(eg. "sql_trace") and then having additional parameters to control
where the output would go. So the sql_trace parameters would be:

* sql_trace_min_planner_duration
* sql_trace_min_executor_duration
* sql_trace_explain_plan

and they would work exactly as you describe, except they would be
settable by normal users.

This is already possible, if your crafty.

Then the destination(s) for the statement
and EXPLAIN logging would be controlled by:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

Just set client_min_messages = 'LOG';

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#14Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Simon Riggs (#13)

Just set client_min_messages = 'LOG';

True, but you would still need to be a superuser to to set the min_durations and
explain parameters. The other advantage of client_sql_trace is that you could
debug your own functions without filling up the log file. It would work better for
multiple users sharing the same DB.

Dean.

_________________________________________________________________
Play and win great prizes with Live Search and Kung Fu Panda
http://clk.atdmt.com/UKM/go/101719966/direct/01/

#15Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Marko Kreen (#12)

Of course you can still sort of see the SQL used in functions declared
SECURITY DEFINER, using debug_print_parse, but this would be opening
up a much more transparent way to do it.

Do I need to worry about this?

Dean

----------------------------------------

Date: Wed, 9 Jul 2008 14:22:45 +0300
From: markokr@gmail.com
To: itagaki.takahiro@oss.ntt.co.jp
Subject: Re: [HACKERS] Auto-explain patch
CC: dean_rasheed@hotmail.com; simon@2ndquadrant.com; pgsql-hackers@postgresql.org

On 7/9/08, ITAGAKI Takahiro wrote:

Dean Rasheed wrote:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

In terms of security, is it ok to show normal users SQLs used in functions
that are owned by other users? Users can call not-owned functions only if
they have EXECUTE privilege on them. -- presently we can see function
bodies from pg_proc.prosrc freely, though.

Different owner is not a problem, but SECURITY DEFINER may be.

That can be solved by turning the setting off on entry to such function,
by non-superuser. Like we handle search_path.

--
marko

_________________________________________________________________
Find the best and worst places on the planet
http://clk.atdmt.com/UKM/go/101719807/direct/01/

#16Marko Kreen
markokr@gmail.com
In reply to: Dean Rasheed (#15)

On 7/9/08, Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Of course you can still sort of see the SQL used in functions declared
SECURITY DEFINER, using debug_print_parse, but this would be opening
up a much more transparent way to do it.

Do I need to worry about this?

If this allows to see values, then yes. Otherwise no.

--
marko

#17Tom Lane
tgl@sss.pgh.pa.us
In reply to: Marko Kreen (#16)

"Marko Kreen" <markokr@gmail.com> writes:

On 7/9/08, Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Do I need to worry about this?

If this allows to see values, then yes. Otherwise no.

It definitely would open a hole that doesn't exist now, which is to
see values that are inserted into an EXECUTE'd query.

regards, tom lane

#18Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#14)

On Wed, 2008-07-09 at 12:01 +0000, Dean Rasheed wrote:

Just set client_min_messages = 'LOG';

True, but you would still need to be a superuser to to set the min_durations and
explain parameters.

No

The other advantage of client_sql_trace is that you could
debug your own functions without filling up the log file. It would work better for
multiple users sharing the same DB.

Not sure about that. We have debuggers and dbms_put_line already.

I think it would be better to introduce a new logging category than to
do something just specifically for this situation.

Suggest new level=TRACE. This allows messages which are not typically
logged by server, yet are specifically requested by client. Admin has
the option to log the messages if desired.

Server log sequence is
DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, INFO, NOTICE, WARNING, ERROR,
*TRACE*, LOG, FATAL, and PANIC.

Rationale for placement: Default is LOG, so TRACE messages not logged by
default. Sometimes set to ERROR. Setting at this position allows full
system trace to be taken if required, then reset when trace complete.

Client log Sequence is
DEBUG5, DEBUG4, DEBUG3, DEBUG2, DEBUG1, LOG, NOTICE, *TRACE*, WARNING,
ERROR, FATAL, and PANIC.

Rationale for placement: many NOTICE messages are generated by various
commands, so many people can still set this off and still get TRACE
messages.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#19Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#10)

On Wed, 2008-07-09 at 09:11 +0000, Dean Rasheed wrote:

Simon, I like your proposal, and I think I can see how to code it
fairly easily.

There is one thing that it doesn't allow, however, which the debug_xxx
parameters do, and that is for a non-superuser to trace SQL used in
functions, from an interactive client session. For me, this is quite a
big thing, because I find it most convienient to turn these parameters
on while writing and tweaking stored procedures, and have the output
go straight to my psql window, without having to connect as a superuser
and trawl through log files.

So I suggest grouping these parameters in their own category
(eg. "sql_trace") and then having additional parameters to control
where the output would go. So the sql_trace parameters would be:

* sql_trace_min_planner_duration
* sql_trace_min_executor_duration
* sql_trace_explain_plan

and they would work exactly as you describe, except they would be
settable by normal users. Then the destination(s) for the statement
and EXPLAIN logging would be controlled by:

* client_sql_trace = on | off - settable by a normal user to allow a
client session to see the sql_trace output. If this parameter is on,
the sql_trace will be logged as NOTICE output.

After sleeping on this, I think we should follow your idea.

If its possible to do the sql_trace_* parameters as a single one, I
would prefer it, since it makes it more practical to use dynamically.
Not sure how...maybe with a wrapper function?

sql_trace(integer) sets just sql_trace_min_executor_duration
sql_trace(integer, boolean) sets executor and explain
sql_trace(integer, integer, boolean) sets all 3

I think you probably need to drop the sql_ off the front because of
parameter length only.

No need for the other log_... parameter though.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#20Gregory Stark
stark@enterprisedb.com
In reply to: Simon Riggs (#19)

"Simon Riggs" <simon@2ndquadrant.com> writes:

On Wed, 2008-07-09 at 09:11 +0000, Dean Rasheed wrote:

So I suggest grouping these parameters in their own category
(eg. "sql_trace") and then having additional parameters to control
where the output would go. So the sql_trace parameters would be:

* sql_trace_min_planner_duration
* sql_trace_min_executor_duration
* sql_trace_explain_plan

If its possible to do the sql_trace_* parameters as a single one, I
would prefer it, since it makes it more practical to use dynamically.
Not sure how...maybe with a wrapper function?

sql_trace(integer) sets just sql_trace_min_executor_duration
sql_trace(integer, boolean) sets executor and explain
sql_trace(integer, integer, boolean) sets all 3

Fwiw it seems to me "trace_sql_*" would be nicer, much as we have
track_* guc parameters.

Though I also wonder if there's really any distinction here between "tracing"
and "logging" like log_explain_plan and so on. Perhaps we should keep the word
"trace" for a more in-detail facility.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

#21Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Gregory Stark (#20)
1 attachment(s)

After sleeping on this, I think we should follow your idea.

Hmm. I preferred your idea ;-) It reduces the number of new parameters
back down to 3, which makes it easier to use:

* trace_min_planner_duration - int, PGC_USERSET
* trace_min_executor_duration - int, PGC_USERSET
* trace_explain_plan - bool, PGC_USERSET

(I already decided to drop the "sql_") with all output going to new level TRACE
as you described. So output goes to client and not server log by default as
soon as either of the first 2 parameters is enabled.

I've attached what I've done so far, which works according to the above
description. I've not done much testing or written any docs yet, It would be
good to know if this is along the right lines.

Changing parameter names is easy, although admittedly very important to
get right. You didn't say why you changed your mind on this?

I'm more concerned about any possible security holes it opens up.
For SQL in SQL functions, it just gives the function name and statement
number. For EXECUTEd queries, it doesn't have access to the SQL, so
it just logs the other stuff and any context info. For plpgsql functions it will
log values which appear as literals in any queries, but actually
debug_print_parse exposes far more info in this case.

_________________________________________________________________
100’s of Nikon cameras to be won with Live Search
http://clk.atdmt.com/UKM/go/101719808/direct/01/

Attachments:

auto-explain3.patchtext/x-patchDownload
*** ./src/backend/commands/copy.c.orig	2008-07-09 16:14:54.000000000 +0100
--- ./src/backend/commands/copy.c	2008-07-09 16:15:54.000000000 +0100
***************
*** 1042,1047 ****
--- 1042,1048 ----
  
  		/* plan the query */
  		plan = planner(query, 0, NULL);
+ 		plan->query_string = queryString;
  
  		/*
  		 * Use a snapshot with an updated command ID to ensure this query sees
*** ./src/backend/commands/explain.c.orig	2008-07-08 13:55:29.000000000 +0100
--- ./src/backend/commands/explain.c	2008-07-09 16:14:28.000000000 +0100
***************
*** 40,65 ****
  explain_get_index_name_hook_type explain_get_index_name_hook = NULL;
  
  
- typedef struct ExplainState
- {
- 	/* options */
- 	bool		printTList;		/* print plan targetlists */
- 	bool		printAnalyze;	/* print actual times */
- 	/* other states */
- 	PlannedStmt *pstmt;			/* top of plan */
- 	List	   *rtable;			/* range table */
- } ExplainState;
- 
  static void ExplainOneQuery(Query *query, ExplainStmt *stmt,
  				const char *queryString,
  				ParamListInfo params, TupOutputState *tstate);
  static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
  				StringInfo buf);
- static double elapsed_time(instr_time *starttime);
- static void explain_outNode(StringInfo str,
- 				Plan *plan, PlanState *planstate,
- 				Plan *outer_plan,
- 				int indent, ExplainState *es);
  static void show_plan_tlist(Plan *plan,
  							StringInfo str, int indent, ExplainState *es);
  static void show_scan_qual(List *qual, const char *qlabel,
--- 40,50 ----
***************
*** 170,175 ****
--- 155,161 ----
  
  		/* plan the query */
  		plan = planner(query, 0, params);
+ 		plan->query_string = queryString;
  
  		/* run it (if needed) and produce output */
  		ExplainOnePlan(plan, params, stmt, tstate);
***************
*** 384,390 ****
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! static double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
--- 370,376 ----
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
***************
*** 406,412 ****
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! static void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
--- 392,398 ----
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
*** ./src/backend/executor/execMain.c.orig	2008-07-08 14:03:34.000000000 +0100
--- ./src/backend/executor/execMain.c	2008-07-10 10:34:26.000000000 +0100
***************
*** 39,44 ****
--- 39,45 ----
  #include "catalog/heap.h"
  #include "catalog/namespace.h"
  #include "catalog/toasting.h"
+ #include "commands/explain.h"
  #include "commands/tablespace.h"
  #include "commands/trigger.h"
  #include "executor/execdebug.h"
***************
*** 52,57 ****
--- 53,59 ----
  #include "storage/lmgr.h"
  #include "storage/smgr.h"
  #include "utils/acl.h"
+ #include "utils/guc.h"
  #include "utils/lsyscache.h"
  #include "utils/memutils.h"
  #include "utils/snapmgr.h"
***************
*** 184,189 ****
--- 186,199 ----
  	}
  
  	/*
+ 	 * If we are tracing and explaining slow-running queries,
+ 	 * enable instrumentation
+ 	 */
+ 	if (trace_explain_plan &&
+ 		(trace_min_planner_duration > -1 || trace_min_executor_duration > -1))
+ 		queryDesc->doInstrument = true;
+ 
+ 	/*
  	 * Copy other important information into the EState
  	 */
  	estate->es_snapshot = RegisterSnapshot(queryDesc->snapshot);
***************
*** 226,231 ****
--- 236,244 ----
  	bool		sendTuples;
  	TupleTableSlot *result;
  	MemoryContext oldcontext;
+ 	bool		time_planner = trace_min_planner_duration > -1;
+ 	bool		time_executor = trace_min_executor_duration > -1;
+ 	instr_time	starttime;
  
  	/* sanity checks */
  	Assert(queryDesc != NULL);
***************
*** 235,240 ****
--- 248,260 ----
  	Assert(estate != NULL);
  
  	/*
+ 	 * If tracing slow-running queries, record the start time before
+ 	 * running the query
+ 	 */
+ 	if (time_executor)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/*
  	 * Switch into per-query memory context
  	 */
  	oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
***************
*** 278,283 ****
--- 298,378 ----
  
  	MemoryContextSwitchTo(oldcontext);
  
+ 	/*
+ 	 * If tracing slow queries, then we should log planner time,
+ 	 * execution time, statement SQL and execution time
+ 	 */
+ 	if (time_planner || time_executor)
+ 	{
+ 		bool log_stmt = false;
+ 		StringInfoData buf;
+ 		double planner_time_ms;
+ 		double executor_time_ms;
+ 
+ 		initStringInfo(&buf);
+ 
+ 		/* Log planner time if appropriate */
+ 		if (time_planner)
+ 		{
+ 			planner_time_ms = queryDesc->plannedstmt->planner_time * 1000.0;
+ 			if (planner_time_ms >= trace_min_planner_duration)
+ 			{
+ 				appendStringInfo(&buf, "planner time: %.3f ms  ",
+ 								 planner_time_ms);
+ 				log_stmt = true;
+ 			}
+ 			/* Don't log again, unless re-planned and still slow */
+ 			queryDesc->plannedstmt->planner_time = 0.0;
+ 		}
+ 
+ 		/* Log execution time if appropriate */
+ 		if (time_executor)
+ 		{
+ 			executor_time_ms = elapsed_time(&starttime) * 1000.0;
+ 			if (executor_time_ms >= trace_min_executor_duration)
+ 			{
+ 				appendStringInfo(&buf, "executor time: %.3f ms  ",
+ 								 executor_time_ms);
+ 				log_stmt = true;
+ 			}
+ 		}
+ 
+ 		/*
+ 		 * Only if either of the above caused the statement to be logged,
+ 		 * then log the statement SQL and maybe its execution plan
+ 		 */
+ 		if (log_stmt)
+ 		{
+ 			if (queryDesc->plannedstmt->query_string)
+ 				appendStringInfo(&buf, "statement: %s",
+ 								 queryDesc->plannedstmt->query_string);
+ 
+ 			if (trace_explain_plan)
+ 			{
+ 				ExplainState *es;
+ 
+ 				es = (ExplainState *) palloc0(sizeof(ExplainState));
+ 
+ 				es->printTList = false;
+ 				es->printAnalyze = true;
+ 				es->pstmt = queryDesc->plannedstmt;
+ 				es->rtable = queryDesc->plannedstmt->rtable;
+ 
+ 				appendStringInfo(&buf, "  plan:\n");
+ 				explain_outNode(&buf,
+ 								queryDesc->plannedstmt->planTree,
+ 								queryDesc->planstate,
+ 								NULL, 0, es);
+ 				pfree(es);
+ 			}
+ 
+ 			/* Finally log this trace report */
+ 			ereport(TRACE, (errmsg(buf.data)));
+ 		}
+ 
+ 		pfree(buf.data);
+ 	}
+ 
  	return result;
  }
  
*** ./src/backend/executor/spi.c.orig	2008-07-09 15:52:10.000000000 +0100
--- ./src/backend/executor/spi.c	2008-07-09 15:58:20.000000000 +0100
***************
*** 1764,1769 ****
--- 1764,1770 ----
  				else
  					snap = InvalidSnapshot;
  
+ 				((PlannedStmt *)stmt)->query_string = plansource->query_string;
  				qdesc = CreateQueryDesc((PlannedStmt *) stmt,
  										snap, crosscheck_snapshot,
  										dest,
*** ./src/backend/optimizer/plan/planner.c.orig	2008-07-09 11:38:06.000000000 +0100
--- ./src/backend/optimizer/plan/planner.c	2008-07-10 10:16:49.000000000 +0100
***************
*** 18,24 ****
--- 18,26 ----
  #include <limits.h>
  
  #include "catalog/pg_operator.h"
+ #include "commands/explain.h"
  #include "executor/executor.h"
+ #include "executor/instrument.h"
  #include "executor/nodeAgg.h"
  #include "miscadmin.h"
  #include "nodes/makefuncs.h"
***************
*** 38,43 ****
--- 40,46 ----
  #include "parser/parse_expr.h"
  #include "parser/parse_oper.h"
  #include "parser/parsetree.h"
+ #include "utils/guc.h"
  #include "utils/lsyscache.h"
  #include "utils/syscache.h"
  
***************
*** 99,109 ****
--- 102,128 ----
  planner(Query *parse, int cursorOptions, ParamListInfo boundParams)
  {
  	PlannedStmt *result;
+ 	instr_time starttime;
  
+ 	/*
+ 	 * If we are tracing statements which take a long time to plan,
+ 	 * then time how long this takes.
+ 	 */
+ 	if (trace_min_planner_duration > -1)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/* Plan the query */
  	if (planner_hook)
  		result = (*planner_hook) (parse, cursorOptions, boundParams);
  	else
  		result = standard_planner(parse, cursorOptions, boundParams);
+ 
+ 	/* Record how long it took, if appropriate */
+ 	if (trace_min_planner_duration)
+ 		result->planner_time = elapsed_time(&starttime);
+ 	else
+ 		result->planner_time = 0.0;
+ 
  	return result;
  }
  
*** ./src/backend/tcop/pquery.c.orig	2008-07-09 16:58:07.000000000 +0100
--- ./src/backend/tcop/pquery.c	2008-07-10 10:33:52.000000000 +0100
***************
*** 895,900 ****
--- 895,903 ----
  	 */
  	queryDesc = PortalGetQueryDesc(portal);
  
+ 	if (queryDesc)
+ 		queryDesc->plannedstmt->query_string = portal->sourceText;
+ 
  	/* Caller messed up if we have neither a ready query nor held data. */
  	Assert(queryDesc || portal->holdStore);
  
***************
*** 1236,1241 ****
--- 1239,1245 ----
  			 * process a plannable query.
  			 */
  			PlannedStmt *pstmt = (PlannedStmt *) stmt;
+ 			pstmt->query_string = portal->sourceText;
  
  			if (log_executor_stats)
  				ResetUsage();
*** ./src/backend/utils/error/elog.c.orig	2008-07-10 09:27:12.000000000 +0100
--- ./src/backend/utils/error/elog.c	2008-07-10 10:09:11.000000000 +0100
***************
*** 1428,1433 ****
--- 1428,1434 ----
  		case COMMERROR:
  		case INFO:
  		case NOTICE:
+ 		case TRACE:
  			eventlevel = EVENTLOG_INFORMATION_TYPE;
  			break;
  		case WARNING:
***************
*** 2027,2032 ****
--- 2028,2034 ----
  				syslog_level = LOG_INFO;
  				break;
  			case NOTICE:
+ 			case TRACE:
  			case WARNING:
  				syslog_level = LOG_NOTICE;
  				break;
***************
*** 2416,2421 ****
--- 2418,2426 ----
  		case NOTICE:
  			prefix = _("NOTICE");
  			break;
+ 		case TRACE:
+ 			prefix = _("TRACE");
+ 			break;
  		case WARNING:
  			prefix = _("WARNING");
  			break;
***************
*** 2506,2511 ****
--- 2511,2519 ----
   * between ERROR and FATAL.  Generally this is the right thing for testing
   * whether a message should go to the postmaster log, whereas a simple >=
   * test is correct for testing whether the message should go to the client.
+  *
+  * Similarly, TRACE is considered to be between ERROR and LOG (<FATAL)
+  * for the purposes of the postmaster log, and so is not output by default.
   */
  static bool
  is_log_level_output(int elevel, int log_min_level)
***************
*** 2517,2523 ****
  	}
  	else if (log_min_level == LOG)
  	{
! 		/* elevel != LOG */
  		if (elevel >= FATAL)
  			return true;
  	}
--- 2525,2543 ----
  	}
  	else if (log_min_level == LOG)
  	{
! 		/* elevel != LOG or COMMERROR */
! 		if (elevel >= FATAL)
! 			return true;
! 	}
! 	else if (elevel == TRACE)
! 	{
! 		/* log_min_level != LOG */
! 		if (log_min_level == TRACE || log_min_level <= ERROR)
! 			return true;
! 	}
! 	else if (log_min_level == TRACE)
! 	{
! 		/* elevel != TRACE, LOG or COMMERROR */
  		if (elevel >= FATAL)
  			return true;
  	}
*** ./src/backend/utils/misc/guc.c.orig	2008-07-09 11:14:31.000000000 +0100
--- ./src/backend/utils/misc/guc.c	2008-07-10 12:01:55.000000000 +0100
***************
*** 187,192 ****
--- 187,193 ----
  	{"log", LOG, false},
  	{"info", INFO, true},
  	{"notice", NOTICE, false},
+ 	{"trace", TRACE, false},
  	{"warning", WARNING, false},
  	{"error", ERROR, false},
  	{"fatal", FATAL, true},
***************
*** 205,210 ****
--- 206,212 ----
  	{"notice", NOTICE, false},
  	{"warning", WARNING, false},
  	{"error", ERROR, false},
+ 	{"trace", TRACE, false},
  	{"log", LOG, false},
  	{"fatal", FATAL, false},
  	{"panic", PANIC, false},
***************
*** 324,329 ****
--- 326,335 ----
  												 * above together */
  bool		log_btree_build_stats = false;
  
+ int			trace_min_planner_duration = -1;
+ int			trace_min_executor_duration = -1;
+ bool		trace_explain_plan = false;
+ 
  bool		check_function_bodies = true;
  bool		default_with_oids = false;
  bool		SQL_inheritance = true;
***************
*** 841,846 ****
--- 847,860 ----
  #endif
  
  	{
+ 		{"trace_explain_plan", PGC_USERSET, LOGGING_WHAT,
+ 			gettext_noop("Enables logging of execution plans for traced statements."),
+ 			NULL
+ 		},
+ 		&trace_explain_plan,
+ 		false, NULL, NULL
+ 	},
+ 	{
  		{"track_activities", PGC_SUSET, STATS_COLLECTOR,
  			gettext_noop("Collects information about executing commands."),
  			gettext_noop("Enables the collection of information on the currently "
***************
*** 1675,1680 ****
--- 1689,1716 ----
  	},
  
  	{
+ 		{"trace_min_planner_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum planner time above which "
+ 						 "statements will be traced."),
+ 			gettext_noop("Zero traces all queries. -1 turns this feature off."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&trace_min_planner_duration,
+ 		-1, -1, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
+ 		{"trace_min_executor_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum execution time above which "
+ 						 "statements will be traced."),
+ 			gettext_noop("Zero traces all queries. -1 turns this feature off."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&trace_min_executor_duration,
+ 		-1, -1, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
  		{"bgwriter_delay", PGC_SIGHUP, RESOURCES,
  			gettext_noop("Background writer sleep time between rounds."),
  			NULL,
*** ./src/include/commands/explain.h.orig	2008-07-08 13:56:55.000000000 +0100
--- ./src/include/commands/explain.h	2008-07-08 14:02:14.000000000 +0100
***************
*** 14,19 ****
--- 14,30 ----
  #define EXPLAIN_H
  
  #include "executor/executor.h"
+ #include "executor/instrument.h"
+ 
+ typedef struct ExplainState
+ {
+ 	/* options */
+ 	bool		printTList;		/* print plan targetlists */
+ 	bool		printAnalyze;	/* print actual times */
+ 	/* other states */
+ 	PlannedStmt *pstmt;			/* top of plan */
+ 	List	   *rtable;			/* range table */
+ } ExplainState;
  
  /* Hook for plugins to get control in ExplainOneQuery() */
  typedef void (*ExplainOneQuery_hook_type) (Query *query,
***************
*** 41,44 ****
--- 52,62 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void explain_outNode(StringInfo str,
+ 				Plan *plan, PlanState *planstate,
+ 				Plan *outer_plan,
+ 				int indent, ExplainState *es);
+ 
+ extern double elapsed_time(instr_time *starttime);
+ 
  #endif   /* EXPLAIN_H */
*** ./src/include/nodes/plannodes.h.orig	2008-07-09 11:34:51.000000000 +0100
--- ./src/include/nodes/plannodes.h	2008-07-10 12:00:02.000000000 +0100
***************
*** 35,40 ****
--- 35,42 ----
  {
  	NodeTag		type;
  
+ 	char		*query_string;	/* the orginal SQL query */
+ 
  	CmdType		commandType;	/* select|insert|update|delete */
  
  	bool		canSetTag;		/* do I set the command result tag? */
***************
*** 73,78 ****
--- 75,86 ----
  	List	   *relationOids;	/* OIDs of relations the plan depends on */
  
  	int			nParamExec;		/* number of PARAM_EXEC Params used */
+ 
+     /*
+ 	 * If tracing statements which are slow to plan, this records the
+ 	 * time taken by the planner (in seconds). Otherwise this will be 0.
+ 	 */
+ 	double		planner_time;
  } PlannedStmt;
  
  /* macro for fetching the Plan associated with a SubPlan node */
*** ./src/include/utils/elog.h.orig	2008-07-10 09:22:47.000000000 +0100
--- ./src/include/utils/elog.h	2008-07-10 09:26:21.000000000 +0100
***************
*** 34,53 ****
  #define NOTICE		18			/* Helpful messages to users about query
  								 * operation;  sent to client and server log
  								 * by default. */
! #define WARNING		19			/* Warnings.  NOTICE is for expected messages
  								 * like implicit sequence creation by SERIAL.
  								 * WARNING is for unexpected messages. */
! #define ERROR		20			/* user error - abort transaction; return to
  								 * known state */
  /* Save ERROR value in PGERROR so it can be restored when Win32 includes
   * modify it.  We have to use a constant rather than ERROR because macros
   * are expanded only when referenced outside macros.
   */
  #ifdef WIN32
! #define PGERROR		20
  #endif
! #define FATAL		21			/* fatal error - abort process */
! #define PANIC		22			/* take down the other backends with me */
  
   /* #define DEBUG DEBUG1 */	/* Backward compatibility with pre-7.3 */
  
--- 34,56 ----
  #define NOTICE		18			/* Helpful messages to users about query
  								 * operation;  sent to client and server log
  								 * by default. */
! #define TRACE		19			/* Trace message to users to help identify
! 								   slow running queries; sent only to the
! 								   client by default. */
! #define WARNING		20			/* Warnings.  NOTICE is for expected messages
  								 * like implicit sequence creation by SERIAL.
  								 * WARNING is for unexpected messages. */
! #define ERROR		21			/* user error - abort transaction; return to
  								 * known state */
  /* Save ERROR value in PGERROR so it can be restored when Win32 includes
   * modify it.  We have to use a constant rather than ERROR because macros
   * are expanded only when referenced outside macros.
   */
  #ifdef WIN32
! #define PGERROR		21
  #endif
! #define FATAL		22			/* fatal error - abort process */
! #define PANIC		23			/* take down the other backends with me */
  
   /* #define DEBUG DEBUG1 */	/* Backward compatibility with pre-7.3 */
  
*** ./src/include/utils/guc.h.orig	2008-07-08 14:07:20.000000000 +0100
--- ./src/include/utils/guc.h	2008-07-10 12:04:20.000000000 +0100
***************
*** 145,150 ****
--- 145,154 ----
  extern int	log_min_duration_statement;
  extern int	log_temp_files;
  
+ extern int	trace_min_planner_duration;
+ extern int	trace_min_executor_duration;
+ extern bool	trace_explain_plan;
+ 
  extern int	num_temp_buffers;
  
  extern char *ConfigFileName;
#22Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Dean Rasheed (#21)
1 attachment(s)

Ooops. That last patch had an embarrassing little typo which caused it to not
actually record the planner times.

This new version fixes that and also includes a little patch to psql so that it
ignores any backend notices during tab-completion, which otherwise just get
in the way. Trace during tab-completion still goes to the server log, if enabled,
since this might actually be useful for debugging psql.

* trace_min_planner_duration - int, PGC_USERSET
* trace_min_executor_duration - int, PGC_USERSET
* trace_explain_plan - bool, PGC_USERSET

I'm toying with the idea of adding another parameter, just for convenience:

* auto_trace = on | off

If trace_min_planner_duration and trace_min_executor_duration are both -1,
then setting auto_trace=on would be equivalent to setting
trace_min_planner_duration=0, trace_min_executor_duration=0 and
trace_explain_plan=on, causing *all* statements to be timed and explained
(similar to Oracle's AUTOTRACE).

If either of trace_min_planner_duration or trace_min_executor_duration
are> -1, then auto_trace will do nothing, and only those queries that are
slow to plan and/or execute would be logged.

I'll hold off actually do any more with this for now though, because I feel
that there are still a couple of questions not fully answered:

* Is a whole new logging level (TRACE) overkill for this, or would it
potentially have other uses in the future? My feeling is that it works quite
nicely.

* It it safe to open this up to ordinary users, or do more restrictions need
to be put on it, and if so what?

Regards, Dean

_________________________________________________________________
The John Lewis Clearance - save up to 50% with FREE delivery
http://clk.atdmt.com/UKM/go/101719806/direct/01/

Attachments:

auto-explain4.patchtext/x-patchDownload
*** ./src/backend/commands/copy.c.orig	2008-07-09 16:14:54.000000000 +0100
--- ./src/backend/commands/copy.c	2008-07-09 16:15:54.000000000 +0100
***************
*** 1042,1047 ****
--- 1042,1048 ----
  
  		/* plan the query */
  		plan = planner(query, 0, NULL);
+ 		plan->query_string = queryString;
  
  		/*
  		 * Use a snapshot with an updated command ID to ensure this query sees
*** ./src/backend/commands/explain.c.orig	2008-07-08 13:55:29.000000000 +0100
--- ./src/backend/commands/explain.c	2008-07-09 16:14:28.000000000 +0100
***************
*** 40,65 ****
  explain_get_index_name_hook_type explain_get_index_name_hook = NULL;
  
  
- typedef struct ExplainState
- {
- 	/* options */
- 	bool		printTList;		/* print plan targetlists */
- 	bool		printAnalyze;	/* print actual times */
- 	/* other states */
- 	PlannedStmt *pstmt;			/* top of plan */
- 	List	   *rtable;			/* range table */
- } ExplainState;
- 
  static void ExplainOneQuery(Query *query, ExplainStmt *stmt,
  				const char *queryString,
  				ParamListInfo params, TupOutputState *tstate);
  static void report_triggers(ResultRelInfo *rInfo, bool show_relname,
  				StringInfo buf);
- static double elapsed_time(instr_time *starttime);
- static void explain_outNode(StringInfo str,
- 				Plan *plan, PlanState *planstate,
- 				Plan *outer_plan,
- 				int indent, ExplainState *es);
  static void show_plan_tlist(Plan *plan,
  							StringInfo str, int indent, ExplainState *es);
  static void show_scan_qual(List *qual, const char *qlabel,
--- 40,50 ----
***************
*** 170,175 ****
--- 155,161 ----
  
  		/* plan the query */
  		plan = planner(query, 0, params);
+ 		plan->query_string = queryString;
  
  		/* run it (if needed) and produce output */
  		ExplainOnePlan(plan, params, stmt, tstate);
***************
*** 384,390 ****
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! static double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
--- 370,376 ----
  }
  
  /* Compute elapsed time in seconds since given timestamp */
! double
  elapsed_time(instr_time *starttime)
  {
  	instr_time	endtime;
***************
*** 406,412 ****
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! static void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
--- 392,398 ----
   * side of a join with the current node.  This is only interesting for
   * deciphering runtime keys of an inner indexscan.
   */
! void
  explain_outNode(StringInfo str,
  				Plan *plan, PlanState *planstate,
  				Plan *outer_plan,
*** ./src/backend/executor/execMain.c.orig	2008-07-08 14:03:34.000000000 +0100
--- ./src/backend/executor/execMain.c	2008-07-10 15:17:31.000000000 +0100
***************
*** 39,44 ****
--- 39,45 ----
  #include "catalog/heap.h"
  #include "catalog/namespace.h"
  #include "catalog/toasting.h"
+ #include "commands/explain.h"
  #include "commands/tablespace.h"
  #include "commands/trigger.h"
  #include "executor/execdebug.h"
***************
*** 52,57 ****
--- 53,59 ----
  #include "storage/lmgr.h"
  #include "storage/smgr.h"
  #include "utils/acl.h"
+ #include "utils/guc.h"
  #include "utils/lsyscache.h"
  #include "utils/memutils.h"
  #include "utils/snapmgr.h"
***************
*** 184,189 ****
--- 186,199 ----
  	}
  
  	/*
+ 	 * If we are tracing and explaining slow-running queries,
+ 	 * enable instrumentation
+ 	 */
+ 	if (trace_explain_plan &&
+ 		(trace_min_planner_duration > -1 || trace_min_executor_duration > -1))
+ 		queryDesc->doInstrument = true;
+ 
+ 	/*
  	 * Copy other important information into the EState
  	 */
  	estate->es_snapshot = RegisterSnapshot(queryDesc->snapshot);
***************
*** 226,231 ****
--- 236,244 ----
  	bool		sendTuples;
  	TupleTableSlot *result;
  	MemoryContext oldcontext;
+ 	bool		time_planner = trace_min_planner_duration > -1;
+ 	bool		time_executor = trace_min_executor_duration > -1;
+ 	instr_time	starttime;
  
  	/* sanity checks */
  	Assert(queryDesc != NULL);
***************
*** 235,240 ****
--- 248,260 ----
  	Assert(estate != NULL);
  
  	/*
+ 	 * If tracing slow-running queries, record the start time before
+ 	 * running the query
+ 	 */
+ 	if (time_executor)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/*
  	 * Switch into per-query memory context
  	 */
  	oldcontext = MemoryContextSwitchTo(estate->es_query_cxt);
***************
*** 278,283 ****
--- 298,378 ----
  
  	MemoryContextSwitchTo(oldcontext);
  
+ 	/*
+ 	 * If tracing slow queries, then we should log planner time,
+ 	 * execution time, statement SQL and execution time
+ 	 */
+ 	if (time_planner || time_executor)
+ 	{
+ 		bool log_stmt = false;
+ 		StringInfoData buf;
+ 		double planner_time_ms;
+ 		double executor_time_ms;
+ 
+ 		initStringInfo(&buf);
+ 
+ 		/* Log planner time if appropriate */
+ 		if (time_planner)
+ 		{
+ 			planner_time_ms = queryDesc->plannedstmt->planner_time * 1000.0;
+ 			if (planner_time_ms >= trace_min_planner_duration)
+ 			{
+ 				appendStringInfo(&buf, "planner time: %.3f ms  ",
+ 								 planner_time_ms);
+ 				log_stmt = true;
+ 			}
+ 			/* Don't log again, unless re-planned and still slow */
+ 			queryDesc->plannedstmt->planner_time = 0.0;
+ 		}
+ 
+ 		/* Log execution time if appropriate */
+ 		if (time_executor)
+ 		{
+ 			executor_time_ms = elapsed_time(&starttime) * 1000.0;
+ 			if (executor_time_ms >= trace_min_executor_duration)
+ 			{
+ 				appendStringInfo(&buf, "executor time: %.3f ms  ",
+ 								 executor_time_ms);
+ 				log_stmt = true;
+ 			}
+ 		}
+ 
+ 		/*
+ 		 * Only if either of the above caused the statement to be logged,
+ 		 * then log the statement SQL and maybe its execution plan
+ 		 */
+ 		if (log_stmt)
+ 		{
+ 			if (queryDesc->plannedstmt->query_string)
+ 				appendStringInfo(&buf, "statement: %s",
+ 								 queryDesc->plannedstmt->query_string);
+ 
+ 			if (trace_explain_plan)
+ 			{
+ 				ExplainState *es;
+ 
+ 				es = (ExplainState *) palloc0(sizeof(ExplainState));
+ 
+ 				es->printTList = false;
+ 				es->printAnalyze = true;
+ 				es->pstmt = queryDesc->plannedstmt;
+ 				es->rtable = queryDesc->plannedstmt->rtable;
+ 
+ 				appendStringInfo(&buf, "  plan:\n");
+ 				explain_outNode(&buf,
+ 								queryDesc->plannedstmt->planTree,
+ 								queryDesc->planstate,
+ 								NULL, 0, es);
+ 				pfree(es);
+ 			}
+ 
+ 			/* Finally log this trace report */
+ 			ereport(TRACE, (errmsg(buf.data)));
+ 		}
+ 
+ 		pfree(buf.data);
+ 	}
+ 
  	return result;
  }
  
*** ./src/backend/executor/spi.c.orig	2008-07-09 15:52:10.000000000 +0100
--- ./src/backend/executor/spi.c	2008-07-09 15:58:20.000000000 +0100
***************
*** 1764,1769 ****
--- 1764,1770 ----
  				else
  					snap = InvalidSnapshot;
  
+ 				((PlannedStmt *)stmt)->query_string = plansource->query_string;
  				qdesc = CreateQueryDesc((PlannedStmt *) stmt,
  										snap, crosscheck_snapshot,
  										dest,
*** ./src/backend/optimizer/plan/planner.c.orig	2008-07-09 11:38:06.000000000 +0100
--- ./src/backend/optimizer/plan/planner.c	2008-07-10 15:17:19.000000000 +0100
***************
*** 18,24 ****
--- 18,26 ----
  #include <limits.h>
  
  #include "catalog/pg_operator.h"
+ #include "commands/explain.h"
  #include "executor/executor.h"
+ #include "executor/instrument.h"
  #include "executor/nodeAgg.h"
  #include "miscadmin.h"
  #include "nodes/makefuncs.h"
***************
*** 38,43 ****
--- 40,46 ----
  #include "parser/parse_expr.h"
  #include "parser/parse_oper.h"
  #include "parser/parsetree.h"
+ #include "utils/guc.h"
  #include "utils/lsyscache.h"
  #include "utils/syscache.h"
  
***************
*** 99,109 ****
--- 102,128 ----
  planner(Query *parse, int cursorOptions, ParamListInfo boundParams)
  {
  	PlannedStmt *result;
+ 	instr_time starttime;
  
+ 	/*
+ 	 * If we are tracing statements which take a long time to plan,
+ 	 * then time how long this takes.
+ 	 */
+ 	if (trace_min_planner_duration > -1)
+ 		INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 	/* Plan the query */
  	if (planner_hook)
  		result = (*planner_hook) (parse, cursorOptions, boundParams);
  	else
  		result = standard_planner(parse, cursorOptions, boundParams);
+ 
+ 	/* Record how long it took, if appropriate */
+ 	if (trace_min_planner_duration > -1)
+ 		result->planner_time = elapsed_time(&starttime);
+ 	else
+ 		result->planner_time = 0.0;
+ 
  	return result;
  }
  
*** ./src/backend/tcop/pquery.c.orig	2008-07-09 16:58:07.000000000 +0100
--- ./src/backend/tcop/pquery.c	2008-07-10 10:33:52.000000000 +0100
***************
*** 895,900 ****
--- 895,903 ----
  	 */
  	queryDesc = PortalGetQueryDesc(portal);
  
+ 	if (queryDesc)
+ 		queryDesc->plannedstmt->query_string = portal->sourceText;
+ 
  	/* Caller messed up if we have neither a ready query nor held data. */
  	Assert(queryDesc || portal->holdStore);
  
***************
*** 1236,1241 ****
--- 1239,1245 ----
  			 * process a plannable query.
  			 */
  			PlannedStmt *pstmt = (PlannedStmt *) stmt;
+ 			pstmt->query_string = portal->sourceText;
  
  			if (log_executor_stats)
  				ResetUsage();
*** ./src/backend/utils/error/elog.c.orig	2008-07-10 09:27:12.000000000 +0100
--- ./src/backend/utils/error/elog.c	2008-07-10 10:09:11.000000000 +0100
***************
*** 1428,1433 ****
--- 1428,1434 ----
  		case COMMERROR:
  		case INFO:
  		case NOTICE:
+ 		case TRACE:
  			eventlevel = EVENTLOG_INFORMATION_TYPE;
  			break;
  		case WARNING:
***************
*** 2027,2032 ****
--- 2028,2034 ----
  				syslog_level = LOG_INFO;
  				break;
  			case NOTICE:
+ 			case TRACE:
  			case WARNING:
  				syslog_level = LOG_NOTICE;
  				break;
***************
*** 2416,2421 ****
--- 2418,2426 ----
  		case NOTICE:
  			prefix = _("NOTICE");
  			break;
+ 		case TRACE:
+ 			prefix = _("TRACE");
+ 			break;
  		case WARNING:
  			prefix = _("WARNING");
  			break;
***************
*** 2506,2511 ****
--- 2511,2519 ----
   * between ERROR and FATAL.  Generally this is the right thing for testing
   * whether a message should go to the postmaster log, whereas a simple >=
   * test is correct for testing whether the message should go to the client.
+  *
+  * Similarly, TRACE is considered to be between ERROR and LOG (<FATAL)
+  * for the purposes of the postmaster log, and so is not output by default.
   */
  static bool
  is_log_level_output(int elevel, int log_min_level)
***************
*** 2517,2523 ****
  	}
  	else if (log_min_level == LOG)
  	{
! 		/* elevel != LOG */
  		if (elevel >= FATAL)
  			return true;
  	}
--- 2525,2543 ----
  	}
  	else if (log_min_level == LOG)
  	{
! 		/* elevel != LOG or COMMERROR */
! 		if (elevel >= FATAL)
! 			return true;
! 	}
! 	else if (elevel == TRACE)
! 	{
! 		/* log_min_level != LOG */
! 		if (log_min_level == TRACE || log_min_level <= ERROR)
! 			return true;
! 	}
! 	else if (log_min_level == TRACE)
! 	{
! 		/* elevel != TRACE, LOG or COMMERROR */
  		if (elevel >= FATAL)
  			return true;
  	}
*** ./src/backend/utils/misc/guc.c.orig	2008-07-09 11:14:31.000000000 +0100
--- ./src/backend/utils/misc/guc.c	2008-07-10 12:01:55.000000000 +0100
***************
*** 187,192 ****
--- 187,193 ----
  	{"log", LOG, false},
  	{"info", INFO, true},
  	{"notice", NOTICE, false},
+ 	{"trace", TRACE, false},
  	{"warning", WARNING, false},
  	{"error", ERROR, false},
  	{"fatal", FATAL, true},
***************
*** 205,210 ****
--- 206,212 ----
  	{"notice", NOTICE, false},
  	{"warning", WARNING, false},
  	{"error", ERROR, false},
+ 	{"trace", TRACE, false},
  	{"log", LOG, false},
  	{"fatal", FATAL, false},
  	{"panic", PANIC, false},
***************
*** 324,329 ****
--- 326,335 ----
  												 * above together */
  bool		log_btree_build_stats = false;
  
+ int			trace_min_planner_duration = -1;
+ int			trace_min_executor_duration = -1;
+ bool		trace_explain_plan = false;
+ 
  bool		check_function_bodies = true;
  bool		default_with_oids = false;
  bool		SQL_inheritance = true;
***************
*** 841,846 ****
--- 847,860 ----
  #endif
  
  	{
+ 		{"trace_explain_plan", PGC_USERSET, LOGGING_WHAT,
+ 			gettext_noop("Enables logging of execution plans for traced statements."),
+ 			NULL
+ 		},
+ 		&trace_explain_plan,
+ 		false, NULL, NULL
+ 	},
+ 	{
  		{"track_activities", PGC_SUSET, STATS_COLLECTOR,
  			gettext_noop("Collects information about executing commands."),
  			gettext_noop("Enables the collection of information on the currently "
***************
*** 1675,1680 ****
--- 1689,1716 ----
  	},
  
  	{
+ 		{"trace_min_planner_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum planner time above which "
+ 						 "statements will be traced."),
+ 			gettext_noop("Zero traces all queries. -1 turns this feature off."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&trace_min_planner_duration,
+ 		-1, -1, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
+ 		{"trace_min_executor_duration", PGC_USERSET, LOGGING_WHEN,
+ 			gettext_noop("Sets the minimum execution time above which "
+ 						 "statements will be traced."),
+ 			gettext_noop("Zero traces all queries. -1 turns this feature off."),
+ 			GUC_UNIT_MS
+ 		},
+ 		&trace_min_executor_duration,
+ 		-1, -1, INT_MAX / 1000, NULL, NULL
+ 	},
+ 
+ 	{
  		{"bgwriter_delay", PGC_SIGHUP, RESOURCES,
  			gettext_noop("Background writer sleep time between rounds."),
  			NULL,
*** ./src/bin/psql/common.c.orig	2008-07-10 15:05:14.000000000 +0100
--- ./src/bin/psql/common.c	2008-07-10 15:05:51.000000000 +0100
***************
*** 26,32 ****
  #include "copy.h"
  #include "mbprint.h"
  
! 
  
  static bool ExecQueryUsingCursor(const char *query, double *elapsed_msec);
  static bool command_no_begin(const char *query);
--- 26,33 ----
  #include "copy.h"
  #include "mbprint.h"
  
! /* Flag to ignore backend notices (used during tab-completion) */
! bool ignore_notices = false;
  
  static bool ExecQueryUsingCursor(const char *query, double *elapsed_msec);
  static bool command_no_begin(const char *query);
***************
*** 180,186 ****
  NoticeProcessor(void *arg, const char *message)
  {
  	(void) arg;					/* not used */
! 	psql_error("%s", message);
  }
  
  
--- 181,189 ----
  NoticeProcessor(void *arg, const char *message)
  {
  	(void) arg;					/* not used */
! 
! 	if (!ignore_notices)
! 		psql_error("%s", message);
  }
  
  
*** ./src/bin/psql/common.h.orig	2008-07-10 15:03:38.000000000 +0100
--- ./src/bin/psql/common.h	2008-07-10 15:04:37.000000000 +0100
***************
*** 63,66 ****
--- 63,69 ----
  
  extern char *expand_tilde(char **filename);
  
+ /* Flag to ignore backend notices (used during tab-completion) */
+ extern bool ignore_notices;
+ 
  #endif   /* COMMON_H */
*** ./src/bin/psql/tab-complete.c.orig	2008-07-10 15:06:22.000000000 +0100
--- ./src/bin/psql/tab-complete.c	2008-07-10 15:06:57.000000000 +0100
***************
*** 2575,2581 ****
--- 2575,2583 ----
  	if (query == NULL || !pset.db || PQstatus(pset.db) != CONNECTION_OK)
  		return NULL;
  
+ 	ignore_notices = true;
  	result = PQexec(pset.db, query);
+ 	ignore_notices = false;
  
  	if (result != NULL && PQresultStatus(result) != PGRES_TUPLES_OK)
  	{
*** ./src/include/commands/explain.h.orig	2008-07-08 13:56:55.000000000 +0100
--- ./src/include/commands/explain.h	2008-07-08 14:02:14.000000000 +0100
***************
*** 14,19 ****
--- 14,30 ----
  #define EXPLAIN_H
  
  #include "executor/executor.h"
+ #include "executor/instrument.h"
+ 
+ typedef struct ExplainState
+ {
+ 	/* options */
+ 	bool		printTList;		/* print plan targetlists */
+ 	bool		printAnalyze;	/* print actual times */
+ 	/* other states */
+ 	PlannedStmt *pstmt;			/* top of plan */
+ 	List	   *rtable;			/* range table */
+ } ExplainState;
  
  /* Hook for plugins to get control in ExplainOneQuery() */
  typedef void (*ExplainOneQuery_hook_type) (Query *query,
***************
*** 41,44 ****
--- 52,62 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void explain_outNode(StringInfo str,
+ 				Plan *plan, PlanState *planstate,
+ 				Plan *outer_plan,
+ 				int indent, ExplainState *es);
+ 
+ extern double elapsed_time(instr_time *starttime);
+ 
  #endif   /* EXPLAIN_H */
*** ./src/include/nodes/plannodes.h.orig	2008-07-09 11:34:51.000000000 +0100
--- ./src/include/nodes/plannodes.h	2008-07-10 12:00:02.000000000 +0100
***************
*** 35,40 ****
--- 35,42 ----
  {
  	NodeTag		type;
  
+ 	char		*query_string;	/* the orginal SQL query */
+ 
  	CmdType		commandType;	/* select|insert|update|delete */
  
  	bool		canSetTag;		/* do I set the command result tag? */
***************
*** 73,78 ****
--- 75,86 ----
  	List	   *relationOids;	/* OIDs of relations the plan depends on */
  
  	int			nParamExec;		/* number of PARAM_EXEC Params used */
+ 
+     /*
+ 	 * If tracing statements which are slow to plan, this records the
+ 	 * time taken by the planner (in seconds). Otherwise this will be 0.
+ 	 */
+ 	double		planner_time;
  } PlannedStmt;
  
  /* macro for fetching the Plan associated with a SubPlan node */
*** ./src/include/utils/elog.h.orig	2008-07-10 09:22:47.000000000 +0100
--- ./src/include/utils/elog.h	2008-07-10 09:26:21.000000000 +0100
***************
*** 34,53 ****
  #define NOTICE		18			/* Helpful messages to users about query
  								 * operation;  sent to client and server log
  								 * by default. */
! #define WARNING		19			/* Warnings.  NOTICE is for expected messages
  								 * like implicit sequence creation by SERIAL.
  								 * WARNING is for unexpected messages. */
! #define ERROR		20			/* user error - abort transaction; return to
  								 * known state */
  /* Save ERROR value in PGERROR so it can be restored when Win32 includes
   * modify it.  We have to use a constant rather than ERROR because macros
   * are expanded only when referenced outside macros.
   */
  #ifdef WIN32
! #define PGERROR		20
  #endif
! #define FATAL		21			/* fatal error - abort process */
! #define PANIC		22			/* take down the other backends with me */
  
   /* #define DEBUG DEBUG1 */	/* Backward compatibility with pre-7.3 */
  
--- 34,56 ----
  #define NOTICE		18			/* Helpful messages to users about query
  								 * operation;  sent to client and server log
  								 * by default. */
! #define TRACE		19			/* Trace message to users to help identify
! 								   slow running queries; sent only to the
! 								   client by default. */
! #define WARNING		20			/* Warnings.  NOTICE is for expected messages
  								 * like implicit sequence creation by SERIAL.
  								 * WARNING is for unexpected messages. */
! #define ERROR		21			/* user error - abort transaction; return to
  								 * known state */
  /* Save ERROR value in PGERROR so it can be restored when Win32 includes
   * modify it.  We have to use a constant rather than ERROR because macros
   * are expanded only when referenced outside macros.
   */
  #ifdef WIN32
! #define PGERROR		21
  #endif
! #define FATAL		22			/* fatal error - abort process */
! #define PANIC		23			/* take down the other backends with me */
  
   /* #define DEBUG DEBUG1 */	/* Backward compatibility with pre-7.3 */
  
*** ./src/include/utils/guc.h.orig	2008-07-08 14:07:20.000000000 +0100
--- ./src/include/utils/guc.h	2008-07-10 12:04:20.000000000 +0100
***************
*** 145,150 ****
--- 145,154 ----
  extern int	log_min_duration_statement;
  extern int	log_temp_files;
  
+ extern int	trace_min_planner_duration;
+ extern int	trace_min_executor_duration;
+ extern bool	trace_explain_plan;
+ 
  extern int	num_temp_buffers;
  
  extern char *ConfigFileName;
#23Simon Riggs
simon@2ndquadrant.com
In reply to: Dean Rasheed (#22)

On Fri, 2008-07-11 at 09:33 +0000, Dean Rasheed wrote:

This new version

Thanks, I'll review this next week now.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#24Tom Lane
tgl@sss.pgh.pa.us
In reply to: Dean Rasheed (#22)

Dean Rasheed <dean_rasheed@hotmail.com> writes:

This new version fixes that and also includes a little patch to psql so that it
ignores any backend notices during tab-completion, which otherwise just get
in the way. Trace during tab-completion still goes to the server log, if enabled,
since this might actually be useful for debugging psql.

Comments:

I do not think that you should invent a new elog level for this, and
especially not one that is designed to send unexpected messages to the
client. Having to kluge tab completion like that is just a signal that
you're going to break a lot of other clients too. It seems to me that
the right behavior for auto-explain messages is to go only to the log by
default, which means that LOG is already a perfectly good elog level for
auto-explain messages.

Drop the query_string addition to PlannedStmt --- there are other ways
you can get that string in CVS HEAD. I don't think that planner_time
belongs there either. It would be impossible to define a principled way
to compare two PlannedStmts for equality with that in there. Nor do I
see the point of doing it the way you're doing it. Why don't you just
log the slow planning cycle immediately upon detecting it in planner()?
I don't see that a slow planning cycle has anything necessarily to do
with a slow execution cycle, so IMHO they ought to just get logged
independently.

Please do not export ExplainState --- that's an internal matter for
explain.c. Export some wrapper function with a cleaner API than
explain_outNode, instead.

regards, tom lane

#25Dean Rasheed
dean_rasheed@hotmail.com
In reply to: Tom Lane (#24)

Thanks for the feedback, and sorry for my delay in replying (I was on
holiday).

Tom Lane wrote:

Comments:

I do not think that you should invent a new elog level for this, and
especially not one that is designed to send unexpected messages to the
client. Having to kluge tab completion like that is just a signal that
you're going to break a lot of other clients too. It seems to me that
the right behavior for auto-explain messages is to go only to the log by
default, which means that LOG is already a perfectly good elog level for
auto-explain messages.

The more I thought about this, the more I thought that it was OTT to
add a new elog level just for this, so I agree it should probably just
go to the LOG elog level.

I don't agree with your reasoning on tab-completion though. I actually
think that this is a signal of a broken client. If the user sets
client_min_messages to LOG or lower, and has any of the other logging
or debugging parameters enabled, the output tramples all over the
suggested completions. I don't think the average user is interested in
log/debug output from the queries psql does internally during
tab-completion. So IMHO the tab-completion 'kludge', is still
worthwhile, regardless of the rest of the patch.

Drop the query_string addition to PlannedStmt --- there are other ways
you can get that string in CVS HEAD.

OK. What is the best way to get this string now? Are you referring to
debug_query_string, or is there another way?

I don't think that planner_time
belongs there either. It would be impossible to define a principled way
to compare two PlannedStmts for equality with that in there. Nor do I
see the point of doing it the way you're doing it. Why don't you just
log the slow planning cycle immediately upon detecting it in planner()?
I don't see that a slow planning cycle has anything necessarily to do
with a slow execution cycle, so IMHO they ought to just get logged
independently.

Makes sense.

Please do not export ExplainState --- that's an internal matter for
explain.c. Export some wrapper function with a cleaner API than
explain_outNode, instead.

regards, tom lane

OK, that's much neater.

I'll try to rework this ASAP but I understand if it's too late for
this commitfest.

Cheers, Dean.

_________________________________________________________________
Win a voice over part with Kung Fu Panda & Live Search   and   100’s of Kung Fu Panda prizes to win with Live Search
http://clk.atdmt.com/UKM/go/107571439/direct/01/

#26ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Dean Rasheed (#25)
1 attachment(s)

Hi,

I'm very interested in the auto-explain feature.
Are there any plans to re-add it in the next commit-fest?

Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Please do not export ExplainState --- that's an internal matter for
explain.c. Export some wrapper function with a cleaner API than
explain_outNode, instead.

OK, that's much neater.

How about the attached patch?
I exported initialization of ExplainState and explain_outNode call
to a new function ExplainOneResult. It receives executor nodes and
returns the tree as a text.

I think it'd be better to implement the auto-explain feature
not as a core feature but as an extension, because various users
have various desires about the feature. We could write a small extension
moudle using hooks and the ExplainOneResult function. If we includes
the extension as a contrib module, users can mofify it and install
customized version of auto-explain.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Attachments:

export_explain.patchapplication/octet-stream; name=export_explain.patchDownload
Index: src/backend/commands/explain.c
===================================================================
--- src/backend/commands/explain.c	(8.4dev 2008-08-26)
+++ src/backend/commands/explain.c	(working copy)
@@ -224,7 +224,6 @@
 	QueryDesc  *queryDesc;
 	instr_time	starttime;
 	double		totaltime = 0;
-	ExplainState *es;
 	StringInfoData buf;
 	int			eflags;
 
@@ -265,17 +264,8 @@
 		totaltime += elapsed_time(&starttime);
 	}
 
-	es = (ExplainState *) palloc0(sizeof(ExplainState));
-
-	es->printTList = stmt->verbose;
-	es->printAnalyze = stmt->analyze;
-	es->pstmt = queryDesc->plannedstmt;
-	es->rtable = queryDesc->plannedstmt->rtable;
-
 	initStringInfo(&buf);
-	explain_outNode(&buf,
-					queryDesc->plannedstmt->planTree, queryDesc->planstate,
-					NULL, 0, es);
+	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
 
 	/*
 	 * If we ran the command, run any AFTER triggers it queued.  (Note this
@@ -290,7 +280,7 @@
 	}
 
 	/* Print info about runtime of triggers */
-	if (es->printAnalyze)
+	if (stmt->analyze)
 	{
 		ResultRelInfo *rInfo;
 		bool		show_relname;
@@ -335,10 +325,29 @@
 	do_text_output_multiline(tstate, buf.data);
 
 	pfree(buf.data);
-	pfree(es);
 }
 
 /*
+ * ExplainOneResult -
+ *	  converts a Plan node into ascii string and appends it to 'str'
+ */
+void
+ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+				 bool analyze, bool verbose)
+{
+	ExplainState	es = { 0 };
+
+	es.printTList = verbose;
+	es.printAnalyze = analyze;
+	es.pstmt = queryDesc->plannedstmt;
+	es.rtable = queryDesc->plannedstmt->rtable;
+
+	explain_outNode(str,
+					queryDesc->plannedstmt->planTree, queryDesc->planstate,
+					NULL, 0, &es);
+}
+
+/*
  * report_triggers -
  *		report execution stats for a single relation's triggers
  */
Index: src/include/commands/explain.h
===================================================================
--- src/include/commands/explain.h	(8.4dev 2008-08-26)
+++ src/include/commands/explain.h	(working copy)
@@ -41,4 +41,7 @@
 extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 			   ExplainStmt *stmt, TupOutputState *tstate);
 
+extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+							 bool analyze, bool verbose);
+
 #endif   /* EXPLAIN_H */
#27Simon Riggs
simon@2ndQuadrant.com
In reply to: ITAGAKI Takahiro (#26)

On Tue, 2008-08-26 at 19:24 +0900, ITAGAKI Takahiro wrote:

I'm very interested in the auto-explain feature.

Me too, though must apologise I've had no further time to review this.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

#28ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: ITAGAKI Takahiro (#26)
4 attachment(s)

Here is a contrib version of auto-explain.
I'd like to add it the next commit-fest in September.

I set a high value on logging, not on interactive responce because
I think it's enough if we use EXPLAIN ANALYZE directly in psql or
set min_client_messages to LOG.

The module consists of one contrib directory and three patches:

* export_explain.patch
It exports an internal routine in explain.c as ExplainOneResult().
Auto-explain module requires it.

* custom_guc_flags.patch
It enables to use guc flags in custom guc variables.
Auto-explain module works better with it because there is a millisecond
unit variable (explain.log_min_duration) in the module.

* psql_ignore_notices.patch
It suppress notice messages during psql tab-completion and
\d commands. I extracted it from Dean's patch.
Auto-explain module does not always need the patch, but I think
this feature is useful even if we don't use auto-explain.
psql will ignore annoying messages on non-user SQLs when we set
min_client_messages to lower level and enable some of log_* or
debug_* options.

* auto_explain.tgz
A contrib module version of auto-explain.
An arguable part is initializing instruments in ExecutorRun_hook.
The initialization should be done in ExecutorStart normally, but
it is too late in the hook. Is it safe? or are there any better idea?
README is a plain-text for now, and I'll rewrite it in sgml if needed.

Comments welcome.

(Here is a copy of README)

auto_explain
------------
Log query plans that execution times are longer than configuration.

Usage
-----
#= LOAD 'auto_explain';
#= SET explain.log_min_duration = 0;
#= SET explain.log_analyze = true;
#= SELECT count(*)
FROM pg_class, pg_index
WHERE oid = indrelid AND indisunique;

LOG: duration: 0.457 ms plan:
Aggregate (cost=14.90..14.91 rows=1 width=0) (actual time=0.444..0.445 rows=1 loops=1)
-> Hash Join (cost=3.91..14.70 rows=81 width=0) (actual time=0.147..0.402 rows=81 loops=1)
Hash Cond: (pg_class.oid = pg_index.indrelid)
-> Seq Scan on pg_class (cost=0.00..8.27 rows=227 width=4) (actual time=0.011..0.135 rows=227 loops=1)
-> Hash (cost=2.90..2.90 rows=81 width=4) (actual time=0.104..0.104 rows=81 loops=1)
-> Seq Scan on pg_index (cost=0.00..2.90 rows=81 width=4) (actual time=0.008..0.056 rows=81 loops=1)
Filter: indisunique
STATEMENT: SELECT count(*)
FROM pg_class, pg_index
WHERE oid = indrelid AND indisunique;

GUC variables
-------------
* explain.log_min_duration (= -1)
Sets the minimum execution time above which plans will be logged.
Zero prints all plans. -1 turns this feature off.

* explain.log_analyze (= false)
Use EXPLAIN ANALYZE for plan logging.

* explain.log_verbose (= false)
Use EXPLAIN VERBOSE for plan logging.

You can use shared_preload_libraries or local_preload_libraries to
load the module automatically. If you do so, you also need to add
"explain" in custom_variable_classes and define explain.* variables
in your postgresql.conf.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Attachments:

custom_guc_flags-0828.patchapplication/octet-stream; name=custom_guc_flags-0828.patchDownload
Index: src/backend/utils/misc/guc.c
===================================================================
--- src/backend/utils/misc/guc.c	(8.4dev 2008-08-28)
+++ src/backend/utils/misc/guc.c	(working copy)
@@ -5588,6 +5588,7 @@
 init_custom_variable(const char *name,
 					 const char *short_desc,
 					 const char *long_desc,
+					 int flags,
 					 GucContext context,
 					 enum config_type type,
 					 size_t sz)
@@ -5602,6 +5603,7 @@
 	gen->group = CUSTOM_OPTIONS;
 	gen->short_desc = short_desc;
 	gen->long_desc = long_desc;
+	gen->flags = flags;
 	gen->vartype = type;
 
 	return gen;
@@ -5677,6 +5679,7 @@
 DefineCustomBoolVariable(const char *name,
 						 const char *short_desc,
 						 const char *long_desc,
+						 int flags,
 						 bool *valueAddr,
 						 GucContext context,
 						 GucBoolAssignHook assign_hook,
@@ -5685,7 +5688,7 @@
 	struct config_bool *var;
 
 	var = (struct config_bool *)
-		init_custom_variable(name, short_desc, long_desc, context,
+		init_custom_variable(name, short_desc, long_desc, flags, context,
 							 PGC_BOOL, sizeof(struct config_bool));
 	var->variable = valueAddr;
 	var->boot_val = *valueAddr;
@@ -5699,6 +5702,7 @@
 DefineCustomIntVariable(const char *name,
 						const char *short_desc,
 						const char *long_desc,
+						int flags,
 						int *valueAddr,
 						int minValue,
 						int maxValue,
@@ -5709,7 +5713,7 @@
 	struct config_int *var;
 
 	var = (struct config_int *)
-		init_custom_variable(name, short_desc, long_desc, context,
+		init_custom_variable(name, short_desc, long_desc, flags, context,
 							 PGC_INT, sizeof(struct config_int));
 	var->variable = valueAddr;
 	var->boot_val = *valueAddr;
@@ -5725,6 +5729,7 @@
 DefineCustomRealVariable(const char *name,
 						 const char *short_desc,
 						 const char *long_desc,
+						 int flags,
 						 double *valueAddr,
 						 double minValue,
 						 double maxValue,
@@ -5735,7 +5740,7 @@
 	struct config_real *var;
 
 	var = (struct config_real *)
-		init_custom_variable(name, short_desc, long_desc, context,
+		init_custom_variable(name, short_desc, long_desc, flags, context,
 							 PGC_REAL, sizeof(struct config_real));
 	var->variable = valueAddr;
 	var->boot_val = *valueAddr;
@@ -5751,6 +5756,7 @@
 DefineCustomStringVariable(const char *name,
 						   const char *short_desc,
 						   const char *long_desc,
+						   int flags,
 						   char **valueAddr,
 						   GucContext context,
 						   GucStringAssignHook assign_hook,
@@ -5759,7 +5765,7 @@
 	struct config_string *var;
 
 	var = (struct config_string *)
-		init_custom_variable(name, short_desc, long_desc, context,
+		init_custom_variable(name, short_desc, long_desc, flags, context,
 							 PGC_STRING, sizeof(struct config_string));
 	var->variable = valueAddr;
 	var->boot_val = *valueAddr;
@@ -5775,6 +5781,7 @@
 DefineCustomEnumVariable(const char *name,
 						 const char *short_desc,
 						 const char *long_desc,
+						 int flags,
 						 int *valueAddr,
 						 const struct config_enum_entry *options,
 						 GucContext context,
@@ -5784,7 +5791,7 @@
 	struct config_enum *var;
 
 	var = (struct config_enum *)
-		init_custom_variable(name, short_desc, long_desc, context,
+		init_custom_variable(name, short_desc, long_desc, flags, context,
 							 PGC_ENUM, sizeof(struct config_enum));
 	var->variable = valueAddr;
 	var->boot_val = *valueAddr;
Index: src/include/utils/guc.h
===================================================================
--- src/include/utils/guc.h	(8.4dev 2008-08-28)
+++ src/include/utils/guc.h	(working copy)
@@ -163,6 +163,7 @@
 						 const char *name,
 						 const char *short_desc,
 						 const char *long_desc,
+						 int flags,
 						 bool *valueAddr,
 						 GucContext context,
 						 GucBoolAssignHook assign_hook,
@@ -172,6 +173,7 @@
 						const char *name,
 						const char *short_desc,
 						const char *long_desc,
+						int flags,
 						int *valueAddr,
 						int minValue,
 						int maxValue,
@@ -183,6 +185,7 @@
 						 const char *name,
 						 const char *short_desc,
 						 const char *long_desc,
+						 int flags,
 						 double *valueAddr,
 						 double minValue,
 						 double maxValue,
@@ -194,6 +197,7 @@
 						   const char *name,
 						   const char *short_desc,
 						   const char *long_desc,
+						   int flags,
 						   char **valueAddr,
 						   GucContext context,
 						   GucStringAssignHook assign_hook,
@@ -203,6 +207,7 @@
 						   const char *name,
 						   const char *short_desc,
 						   const char *long_desc,
+						   int flags,
 						   int *valueAddr,
 						   const struct config_enum_entry *options,
 						   GucContext context,
export_explain.patchapplication/octet-stream; name=export_explain.patchDownload
Index: src/backend/commands/explain.c
===================================================================
--- src/backend/commands/explain.c	(8.4dev 2008-08-26)
+++ src/backend/commands/explain.c	(working copy)
@@ -224,7 +224,6 @@
 	QueryDesc  *queryDesc;
 	instr_time	starttime;
 	double		totaltime = 0;
-	ExplainState *es;
 	StringInfoData buf;
 	int			eflags;
 
@@ -265,17 +264,8 @@
 		totaltime += elapsed_time(&starttime);
 	}
 
-	es = (ExplainState *) palloc0(sizeof(ExplainState));
-
-	es->printTList = stmt->verbose;
-	es->printAnalyze = stmt->analyze;
-	es->pstmt = queryDesc->plannedstmt;
-	es->rtable = queryDesc->plannedstmt->rtable;
-
 	initStringInfo(&buf);
-	explain_outNode(&buf,
-					queryDesc->plannedstmt->planTree, queryDesc->planstate,
-					NULL, 0, es);
+	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
 
 	/*
 	 * If we ran the command, run any AFTER triggers it queued.  (Note this
@@ -290,7 +280,7 @@
 	}
 
 	/* Print info about runtime of triggers */
-	if (es->printAnalyze)
+	if (stmt->analyze)
 	{
 		ResultRelInfo *rInfo;
 		bool		show_relname;
@@ -335,10 +325,29 @@
 	do_text_output_multiline(tstate, buf.data);
 
 	pfree(buf.data);
-	pfree(es);
 }
 
 /*
+ * ExplainOneResult -
+ *	  converts a Plan node into ascii string and appends it to 'str'
+ */
+void
+ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+				 bool analyze, bool verbose)
+{
+	ExplainState	es = { 0 };
+
+	es.printTList = verbose;
+	es.printAnalyze = analyze;
+	es.pstmt = queryDesc->plannedstmt;
+	es.rtable = queryDesc->plannedstmt->rtable;
+
+	explain_outNode(str,
+					queryDesc->plannedstmt->planTree, queryDesc->planstate,
+					NULL, 0, &es);
+}
+
+/*
  * report_triggers -
  *		report execution stats for a single relation's triggers
  */
Index: src/include/commands/explain.h
===================================================================
--- src/include/commands/explain.h	(8.4dev 2008-08-26)
+++ src/include/commands/explain.h	(working copy)
@@ -41,4 +41,7 @@
 extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 			   ExplainStmt *stmt, TupOutputState *tstate);
 
+extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+							 bool analyze, bool verbose);
+
 #endif   /* EXPLAIN_H */
psql_ignore_notices-0828.patchapplication/octet-stream; name=psql_ignore_notices-0828.patchDownload
Index: src/bin/psql/command.c
===================================================================
--- src/bin/psql/command.c	(8.4dev 2008-08-28)
+++ src/bin/psql/command.c	(working copy)
@@ -333,6 +333,7 @@
 
 		show_verbose = strchr(cmd, '+') ? true : false;
 
+		ignore_notices = true;
 		switch (cmd[1])
 		{
 			case '\0':
@@ -418,6 +419,7 @@
 			default:
 				status = PSQL_CMD_UNKNOWN;
 		}
+		ignore_notices = false;
 
 		if (pattern)
 			free(pattern);
Index: src/bin/psql/common.c
===================================================================
--- src/bin/psql/common.c	(8.4dev 2008-08-28)
+++ src/bin/psql/common.c	(working copy)
@@ -26,8 +26,9 @@
 #include "copy.h"
 #include "mbprint.h"
 
+/* Flag to ignore backend notices (used during tab-completion) */
+bool ignore_notices = false;
 
-
 static bool ExecQueryUsingCursor(const char *query, double *elapsed_msec);
 static bool command_no_begin(const char *query);
 static bool is_select_command(const char *query);
@@ -180,7 +181,9 @@
 NoticeProcessor(void *arg, const char *message)
 {
 	(void) arg;					/* not used */
-	psql_error("%s", message);
+
+	if (!ignore_notices)
+		psql_error("%s", message);
 }
 
 
Index: src/bin/psql/common.h
===================================================================
--- src/bin/psql/common.h	(8.4dev 2008-08-28)
+++ src/bin/psql/common.h	(working copy)
@@ -63,4 +63,7 @@
 
 extern char *expand_tilde(char **filename);
 
+/* Flag to ignore backend notices (used during tab-completion) */
+extern bool ignore_notices;
+
 #endif   /* COMMON_H */
Index: src/bin/psql/tab-complete.c
===================================================================
--- src/bin/psql/tab-complete.c	(8.4dev 2008-08-28)
+++ src/bin/psql/tab-complete.c	(working copy)
@@ -2575,7 +2575,9 @@
 	if (query == NULL || !pset.db || PQstatus(pset.db) != CONNECTION_OK)
 		return NULL;
 
+	ignore_notices = true;
 	result = PQexec(pset.db, query);
+	ignore_notices = false;
 
 	if (PQresultStatus(result) != PGRES_TUPLES_OK)
 	{
auto_explain-0828.tgzapplication/octet-stream; name=auto_explain-0828.tgzDownload
#29Dean Rasheed
dean_rasheed@hotmail.com
In reply to: ITAGAKI Takahiro (#28)

Here is a contrib version of auto-explain.

OK, I hadn't considered doing it as a module before.

Is it only available to superusers? Do we have a general policy on
this? Most logging options are superuser-only, but the recent changes
to LOG debug_print_* output have left them PGC_USERSET.

Regards, Dean.

_________________________________________________________________
Win a voice over part with Kung Fu Panda & Live Search   and   100’s of Kung Fu Panda prizes to win with Live Search
http://clk.atdmt.com/UKM/go/107571439/direct/01/

#30ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Dean Rasheed (#29)

Dean Rasheed <dean_rasheed@hotmail.com> wrote:

Is it only available to superusers?

Presently, yes.

Do we have a general policy on
this? Most logging options are superuser-only, but the recent changes
to LOG debug_print_* output have left them PGC_USERSET.

I set it PGC_SUSET because other log_* options have PGC_SUSET.
I'm not sure what is the difference between log_* and debug_* ...

I think the best policy is to allow general users only to turn "on"
those options, but not to turn "off" if the default setting is on.
We might need to get default values in postgresq.conf in the context
of assignment.

Here is a psesudo code. Is it possible?

bool log_xxx_assign_hook(newval, source)
{
if (by SET command && !superuser())
{
if (default_value == true && newval == false)
elog(ERROR, "You cannot turn off the loggging option");
}
return true; /* ok, you can reset it */
}

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#31Dean Rasheed
dean_rasheed@hotmail.com
In reply to: ITAGAKI Takahiro (#28)

* auto_explain.tgz
A contrib module version of auto-explain.
An arguable part is initializing instruments in ExecutorRun_hook.
The initialization should be done in ExecutorStart normally, but
it is too late in the hook. Is it safe? or are there any better idea?
README is a plain-text for now, and I'll rewrite it in sgml if needed.

How about adding a new hook to control instrumentation of queries in
ExecutorStart? Something like:

typedef bool (*ExecutorDoInstrument_hook_type) (QueryDesc *queryDesc, int eflags);

extern PGDLLIMPORT ExecutorDoInstrument_hook_type ExecutorDoInstrument_hook;

I think that would allow your module code to be simplified,
and it would allow other future extensions to hook in and
enable instrumentation before queries are run.

Regards, Dean.

_________________________________________________________________
Win a voice over part with Kung Fu Panda & Live Search   and   100’s of Kung Fu Panda prizes to win with Live Search
http://clk.atdmt.com/UKM/go/107571439/direct/01/

#32ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Dean Rasheed (#31)

Dean Rasheed <dean_rasheed@hotmail.com> wrote:

An arguable part is initializing instruments in ExecutorRun_hook.
The initialization should be done in ExecutorStart normally, but
it is too late in the hook. Is it safe? or are there any better idea?

How about adding a new hook to control instrumentation of queries in
ExecutorStart? Something like:

typedef bool (*ExecutorDoInstrument_hook_type) (QueryDesc *queryDesc, int eflags);
extern PGDLLIMPORT ExecutorDoInstrument_hook_type ExecutorDoInstrument_hook;

I think it is not good to have any single-purpose hooks -- a new global
variable "bool force_instrument" would be enough for the purpose ;-)

I'd like to suggest on-demand allocation of instruments instead.
PlanState->instrument is not only a runtime statstics collector, but also
represents whether instrumentation is enabled or not. However, we also
have the same information in EState->es_insrument. If we use it instread
of NULL check, we could initialize instrument fields in executor.

[src/backend/executor/execProcnode.c]
ExecProcNode(PlanState *node)
{
...
if (node->state->es_instrument)
{
if (node->instrument == NULL)
node->instrument = InstrAlloc(1, long_lived_memory_context);
InstrStartNode(node->instrument);
}

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#33Marko Kreen
markokr@gmail.com
In reply to: ITAGAKI Takahiro (#28)

On 8/28/08, ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:

Here is a contrib version of auto-explain.

You can use shared_preload_libraries or local_preload_libraries to
load the module automatically. If you do so, you also need to add
"explain" in custom_variable_classes and define explain.* variables
in your postgresql.conf.

Is it possible to use LOAD command to load the module?

--
marko

#34ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Marko Kreen (#33)

"Marko Kreen" <markokr@gmail.com> wrote:

On 8/28/08, ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:

You can use shared_preload_libraries or local_preload_libraries to
load the module automatically. If you do so, you also need to add
"explain" in custom_variable_classes and define explain.* variables
in your postgresql.conf.

Is it possible to use LOAD command to load the module?

Yes, but disabled in default.
You need to enable it like:

LOAD 'auto_explain';
SET explain.log_min_duration = '100ms';
SET explain.log_analyze = true;
SELECT ...

In that case, you don't need to define custom_variable_classes.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#35Marko Kreen
markokr@gmail.com
In reply to: ITAGAKI Takahiro (#34)

On 9/2/08, ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:

"Marko Kreen" <markokr@gmail.com> wrote:

On 8/28/08, ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:

You can use shared_preload_libraries or local_preload_libraries to
load the module automatically. If you do so, you also need to add
"explain" in custom_variable_classes and define explain.* variables
in your postgresql.conf.

Is it possible to use LOAD command to load the module?

Yes, but disabled in default.
You need to enable it like:

LOAD 'auto_explain';
SET explain.log_min_duration = '100ms';

SET explain.log_analyze = true;

SELECT ...

In that case, you don't need to define custom_variable_classes.

I was interested if it is possible to enable it for single session.
Seems it is. Good.

--
marko

#36Alex Hunsaker
badalex@gmail.com
In reply to: ITAGAKI Takahiro (#28)
1 attachment(s)

On Wed, Aug 27, 2008 at 8:54 PM, ITAGAKI Takahiro
<itagaki.takahiro@oss.ntt.co.jp> wrote:

Here is a contrib version of auto-explain.
I'd like to add it the next commit-fest in September.

Here is my review:

*custom_guc_flags-0828.patch

It seems to be windows newline damaged? lots of ^M at the end of the
lines, could not get it to apply cleanly. New patch attached.

My only other concern is the changes to DefineCustom*() to tag the new
flags param. Now I think anyone who uses Custom gucs will want/should
be able to set that. I did not see any people in contrib using it but
did not look on PGfoundry. Do we need to document the change
somewhere for people who might be using it???

*export_explain.patch:

This looks much better than the old exporting of ExplainState way and
fixes tom's complaint about exporting ExplainState, so looks good to
me.

*psql_ignore_notices-0828.patch:

This is not needed anymore because we log to LOG. (as you pointed out)
Tom also voiced some concern about this possibly breaking (or broken) clients.

I set my client_min_messages to notice and tried tab completing common
things I do.. I did not see any NOTICES, so unless we have an annoying
message someone knows about (and maybe it should not be a notice
then). I think this should get dropped.

*auto_explalin.c:

init_instrument()

Hrm this strikes me as kind of kludgy... Any thoughts on the 4 cases
in the switch getting out of sync (T_AppendState, T_BitmapAndState,
T_BitmapOrState, T_SubqueryScanState)? The only "cleaner" way I can
see is to add a hook for CreateQueryDesc so we can overload
doInstrument and ExecInitNode will InstrAlloc them all for us.
I dunno Suggestions??

use the {outer|inner}PlanState macros instead of ->lefttree, ->righttree

can probably save a few cycles by wrapping those in a
if (outerPlanNode(planstate))

A quick check shows they can be null, but maybe they never can be when
they get to this point... So maybe thats a mute point.

If this sticks around I would suggest restructuring it to better match
explain_outNode so its easier to match up
something like...

if (planstate->initPlan)
foreach...
init_instrument()

if (outerPlanState())
foreach...

if (innerPlanState())

the only other comment I have is suset_assign() do we really need to
be a superuser if its all going to LOG ? There was some concern about
explaining security definer functions right? but surely a regular
explain on those shows the same thing as this explain? Or what am I
missing?

and obviously your self noted comment that README.txt should be sgml

Attachments:

custom_guc_flags.patchapplication/octet-stream; name=custom_guc_flags.patchDownload
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 5619,5624 **** static struct config_generic *
--- 5619,5625 ----
  init_custom_variable(const char *name,
  					 const char *short_desc,
  					 const char *long_desc,
+ 					 int flags,
  					 GucContext context,
  					 enum config_type type,
  					 size_t sz)
***************
*** 5633,5638 **** init_custom_variable(const char *name,
--- 5634,5640 ----
  	gen->group = CUSTOM_OPTIONS;
  	gen->short_desc = short_desc;
  	gen->long_desc = long_desc;
+ 	gen->flags = flags;
  	gen->vartype = type;
  
  	return gen;
***************
*** 5708,5713 **** void
--- 5710,5716 ----
  DefineCustomBoolVariable(const char *name,
  						 const char *short_desc,
  						 const char *long_desc,
+ 						 int flags,
  						 bool *valueAddr,
  						 GucContext context,
  						 GucBoolAssignHook assign_hook,
***************
*** 5716,5722 **** DefineCustomBoolVariable(const char *name,
  	struct config_bool *var;
  
  	var = (struct config_bool *)
! 		init_custom_variable(name, short_desc, long_desc, context,
  							 PGC_BOOL, sizeof(struct config_bool));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
--- 5719,5725 ----
  	struct config_bool *var;
  
  	var = (struct config_bool *)
! 		init_custom_variable(name, short_desc, long_desc, flags, context,
  							 PGC_BOOL, sizeof(struct config_bool));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
***************
*** 5730,5735 **** void
--- 5733,5739 ----
  DefineCustomIntVariable(const char *name,
  						const char *short_desc,
  						const char *long_desc,
+ 						int flags,
  						int *valueAddr,
  						int minValue,
  						int maxValue,
***************
*** 5740,5746 **** DefineCustomIntVariable(const char *name,
  	struct config_int *var;
  
  	var = (struct config_int *)
! 		init_custom_variable(name, short_desc, long_desc, context,
  							 PGC_INT, sizeof(struct config_int));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
--- 5744,5750 ----
  	struct config_int *var;
  
  	var = (struct config_int *)
! 		init_custom_variable(name, short_desc, long_desc, flags, context,
  							 PGC_INT, sizeof(struct config_int));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
***************
*** 5756,5761 **** void
--- 5760,5766 ----
  DefineCustomRealVariable(const char *name,
  						 const char *short_desc,
  						 const char *long_desc,
+ 						 int flags,
  						 double *valueAddr,
  						 double minValue,
  						 double maxValue,
***************
*** 5766,5772 **** DefineCustomRealVariable(const char *name,
  	struct config_real *var;
  
  	var = (struct config_real *)
! 		init_custom_variable(name, short_desc, long_desc, context,
  							 PGC_REAL, sizeof(struct config_real));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
--- 5771,5777 ----
  	struct config_real *var;
  
  	var = (struct config_real *)
! 		init_custom_variable(name, short_desc, long_desc, flags, context,
  							 PGC_REAL, sizeof(struct config_real));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
***************
*** 5782,5787 **** void
--- 5787,5793 ----
  DefineCustomStringVariable(const char *name,
  						   const char *short_desc,
  						   const char *long_desc,
+ 						   int flags,
  						   char **valueAddr,
  						   GucContext context,
  						   GucStringAssignHook assign_hook,
***************
*** 5790,5796 **** DefineCustomStringVariable(const char *name,
  	struct config_string *var;
  
  	var = (struct config_string *)
! 		init_custom_variable(name, short_desc, long_desc, context,
  							 PGC_STRING, sizeof(struct config_string));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
--- 5796,5802 ----
  	struct config_string *var;
  
  	var = (struct config_string *)
! 		init_custom_variable(name, short_desc, long_desc, flags, context,
  							 PGC_STRING, sizeof(struct config_string));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
***************
*** 5806,5811 **** void
--- 5812,5818 ----
  DefineCustomEnumVariable(const char *name,
  						 const char *short_desc,
  						 const char *long_desc,
+ 						 int flags,
  						 int *valueAddr,
  						 const struct config_enum_entry *options,
  						 GucContext context,
***************
*** 5815,5821 **** DefineCustomEnumVariable(const char *name,
  	struct config_enum *var;
  
  	var = (struct config_enum *)
! 		init_custom_variable(name, short_desc, long_desc, context,
  							 PGC_ENUM, sizeof(struct config_enum));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
--- 5822,5828 ----
  	struct config_enum *var;
  
  	var = (struct config_enum *)
! 		init_custom_variable(name, short_desc, long_desc, flags, context,
  							 PGC_ENUM, sizeof(struct config_enum));
  	var->variable = valueAddr;
  	var->boot_val = *valueAddr;
*** a/src/include/utils/guc.h
--- b/src/include/utils/guc.h
***************
*** 163,168 **** extern void DefineCustomBoolVariable(
--- 163,169 ----
  						 const char *name,
  						 const char *short_desc,
  						 const char *long_desc,
+ 						 int flags,
  						 bool *valueAddr,
  						 GucContext context,
  						 GucBoolAssignHook assign_hook,
***************
*** 172,177 **** extern void DefineCustomIntVariable(
--- 173,179 ----
  						const char *name,
  						const char *short_desc,
  						const char *long_desc,
+ 						int flags,
  						int *valueAddr,
  						int minValue,
  						int maxValue,
***************
*** 183,188 **** extern void DefineCustomRealVariable(
--- 185,191 ----
  						 const char *name,
  						 const char *short_desc,
  						 const char *long_desc,
+ 						 int flags,
  						 double *valueAddr,
  						 double minValue,
  						 double maxValue,
***************
*** 194,199 **** extern void DefineCustomStringVariable(
--- 197,203 ----
  						   const char *name,
  						   const char *short_desc,
  						   const char *long_desc,
+ 						   int flags,
  						   char **valueAddr,
  						   GucContext context,
  						   GucStringAssignHook assign_hook,
***************
*** 203,208 **** extern void DefineCustomEnumVariable(
--- 207,213 ----
  						   const char *name,
  						   const char *short_desc,
  						   const char *long_desc,
+ 						   int flags,
  						   int *valueAddr,
  						   const struct config_enum_entry *options,
  						   GucContext context,
#37ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Alex Hunsaker (#36)
3 attachment(s)
auto_explain contrib moudle

Thanks for your reviewing, Alex.
I applied your comments to my patch.

- auto_explain.patch : patch against HEAD
- auto_explain.tgz : contrib module
- autoexplain.sgml : documentation

"Alex Hunsaker" <badalex@gmail.com> wrote:

*custom_guc_flags-0828.patch
My only other concern is the changes to DefineCustom*() to tag the new
flags param. Now I think anyone who uses Custom gucs will want/should
be able to set that. I did not see any people in contrib using it but
did not look on PGfoundry. Do we need to document the change
somewhere for people who might be using it???

Now it is done with DefineCustomVariable(type, variable) and keep
existing functions as-is for backward compatibility.

Some people will be happy if the functions are documented,
but we need to define 'stable-internal-functions' between
SPI (stable expoted functions) and unstable internal functions.

*psql_ignore_notices-0828.patch:
I think this should get dropped.

Hmm, I agree that hiding all messages is not good. I removed it.
If some people need it, we can reconsider it in a separated discussion.

*auto_explalin.c:
init_instrument()
The only "cleaner" way I can
see is to add a hook for CreateQueryDesc so we can overload
doInstrument and ExecInitNode will InstrAlloc them all for us.

I wanted to avoid modifying core codes as far as possible,
but I see it was ugly. Now I added 'force_instrument' global
variable as a hook for CreateQueryDesc.

the only other comment I have is suset_assign() do we really need to
be a superuser if its all going to LOG ? There was some concern about
explaining security definer functions right? but surely a regular
explain on those shows the same thing as this explain? Or what am I
missing?

Almost logging options in postgres are only for superusers. So I think
auto_explain options should not be modified by non-superusers, too.

If you want to permit usage for users, you can create a security definer
wrapper function to allow it, no?

CREATE FUNCTION set_explain_log_min_duration(text) RETURNS text AS
$$ SELECT set_config('explain.log_min_duration', $1, false); $$
LANGUAGE sql SECURITY DEFINER;

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Attachments:

auto_explain.patchapplication/octet-stream; name=auto_explain.patchDownload
diff -cpr HEAD/src/backend/commands/explain.c auto_explain/src/backend/commands/explain.c
*** HEAD/src/backend/commands/explain.c	Tue Oct  7 05:29:38 2008
--- auto_explain/src/backend/commands/explain.c	Thu Oct  9 18:10:23 2008
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 224,230 ****
  	QueryDesc  *queryDesc;
  	instr_time	starttime;
  	double		totaltime = 0;
- 	ExplainState *es;
  	StringInfoData buf;
  	int			eflags;
  
--- 224,229 ----
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 265,281 ****
  		totaltime += elapsed_time(&starttime);
  	}
  
- 	es = (ExplainState *) palloc0(sizeof(ExplainState));
- 
- 	es->printTList = stmt->verbose;
- 	es->printAnalyze = stmt->analyze;
- 	es->pstmt = queryDesc->plannedstmt;
- 	es->rtable = queryDesc->plannedstmt->rtable;
- 
  	initStringInfo(&buf);
! 	explain_outNode(&buf,
! 					queryDesc->plannedstmt->planTree, queryDesc->planstate,
! 					NULL, 0, es);
  
  	/*
  	 * If we ran the command, run any AFTER triggers it queued.  (Note this
--- 264,271 ----
  		totaltime += elapsed_time(&starttime);
  	}
  
  	initStringInfo(&buf);
! 	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
  
  	/*
  	 * If we ran the command, run any AFTER triggers it queued.  (Note this
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 290,296 ****
  	}
  
  	/* Print info about runtime of triggers */
! 	if (es->printAnalyze)
  	{
  		ResultRelInfo *rInfo;
  		bool		show_relname;
--- 280,286 ----
  	}
  
  	/* Print info about runtime of triggers */
! 	if (stmt->analyze)
  	{
  		ResultRelInfo *rInfo;
  		bool		show_relname;
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 335,341 ****
  	do_text_output_multiline(tstate, buf.data);
  
  	pfree(buf.data);
! 	pfree(es);
  }
  
  /*
--- 325,350 ----
  	do_text_output_multiline(tstate, buf.data);
  
  	pfree(buf.data);
! }
! 
! /*
!  * ExplainOneResult -
!  *	  converts a Plan node into ascii string and appends it to 'str'
!  */
! void
! ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
! 				 bool analyze, bool verbose)
! {
! 	ExplainState	es = { 0 };
! 
! 	es.printTList = verbose;
! 	es.printAnalyze = analyze;
! 	es.pstmt = queryDesc->plannedstmt;
! 	es.rtable = queryDesc->plannedstmt->rtable;
! 
! 	explain_outNode(str,
! 					queryDesc->plannedstmt->planTree, queryDesc->planstate,
! 					NULL, 0, &es);
  }
  
  /*
diff -cpr HEAD/src/backend/tcop/pquery.c auto_explain/src/backend/tcop/pquery.c
*** HEAD/src/backend/tcop/pquery.c	Fri Aug  1 22:16:09 2008
--- auto_explain/src/backend/tcop/pquery.c	Thu Oct  9 18:10:23 2008
***************
*** 32,38 ****
   * if there are several).
   */
  Portal		ActivePortal = NULL;
! 
  
  static void ProcessQuery(PlannedStmt *plan,
  			 ParamListInfo params,
--- 32,38 ----
   * if there are several).
   */
  Portal		ActivePortal = NULL;
! bool		force_instrument = false;
  
  static void ProcessQuery(PlannedStmt *plan,
  			 ParamListInfo params,
*************** CreateQueryDesc(PlannedStmt *plannedstmt
*** 76,82 ****
  	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
  	qd->dest = dest;			/* output dest */
  	qd->params = params;		/* parameter values passed into query */
! 	qd->doInstrument = doInstrument;	/* instrumentation wanted? */
  
  	/* null these fields until set by ExecutorStart */
  	qd->tupDesc = NULL;
--- 76,82 ----
  	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
  	qd->dest = dest;			/* output dest */
  	qd->params = params;		/* parameter values passed into query */
! 	qd->doInstrument = force_instrument || doInstrument;	/* instrumentation wanted? */
  
  	/* null these fields until set by ExecutorStart */
  	qd->tupDesc = NULL;
diff -cpr HEAD/src/backend/utils/misc/guc.c auto_explain/src/backend/utils/misc/guc.c
*** HEAD/src/backend/utils/misc/guc.c	Mon Oct  6 22:05:36 2008
--- auto_explain/src/backend/utils/misc/guc.c	Thu Oct  9 18:10:23 2008
*************** static const char *assign_pgstat_temp_di
*** 169,174 ****
--- 169,175 ----
  
  static char *config_enum_get_options(struct config_enum *record, 
  									 const char *prefix, const char *suffix);
+ static void initialize_option(struct config_generic *gconf);
  
  
  /*
*************** InitializeGUCOptions(void)
*** 3177,3288 ****
  	for (i = 0; i < num_guc_variables; i++)
  	{
  		struct config_generic *gconf = guc_variables[i];
! 
! 		gconf->status = 0;
! 		gconf->reset_source = PGC_S_DEFAULT;
! 		gconf->source = PGC_S_DEFAULT;
! 		gconf->stack = NULL;
! 		gconf->sourcefile = NULL;
! 		gconf->sourceline = 0;
! 
! 		switch (gconf->vartype)
! 		{
! 			case PGC_BOOL:
! 				{
! 					struct config_bool *conf = (struct config_bool *) gconf;
! 
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %d",
! 								 conf->gen.name, (int) conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_INT:
! 				{
! 					struct config_int *conf = (struct config_int *) gconf;
! 
! 					Assert(conf->boot_val >= conf->min);
! 					Assert(conf->boot_val <= conf->max);
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %d",
! 								 conf->gen.name, conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_REAL:
! 				{
! 					struct config_real *conf = (struct config_real *) gconf;
! 
! 					Assert(conf->boot_val >= conf->min);
! 					Assert(conf->boot_val <= conf->max);
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %g",
! 								 conf->gen.name, conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_STRING:
! 				{
! 					struct config_string *conf = (struct config_string *) gconf;
! 					char	   *str;
! 
! 					*conf->variable = NULL;
! 					conf->reset_val = NULL;
! 
! 					if (conf->boot_val == NULL)
! 					{
! 						/* leave the value NULL, do not call assign hook */
! 						break;
! 					}
! 
! 					str = guc_strdup(FATAL, conf->boot_val);
! 					conf->reset_val = str;
! 
! 					if (conf->assign_hook)
! 					{
! 						const char *newstr;
! 
! 						newstr = (*conf->assign_hook) (str, true,
! 													   PGC_S_DEFAULT);
! 						if (newstr == NULL)
! 						{
! 							elog(FATAL, "failed to initialize %s to \"%s\"",
! 								 conf->gen.name, str);
! 						}
! 						else if (newstr != str)
! 						{
! 							free(str);
! 
! 							/*
! 							 * See notes in set_config_option about casting
! 							 */
! 							str = (char *) newstr;
! 							conf->reset_val = str;
! 						}
! 					}
! 					*conf->variable = str;
! 					break;
! 				}
! 			case PGC_ENUM:
! 				{
! 					struct config_enum *conf = (struct config_enum *) gconf;
! 
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %s",
! 								 conf->gen.name, 
! 								 config_enum_lookup_by_value(conf, conf->boot_val));
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 		}
  	}
  
  	guc_dirty = false;
--- 3178,3184 ----
  	for (i = 0; i < num_guc_variables; i++)
  	{
  		struct config_generic *gconf = guc_variables[i];
! 		initialize_option(gconf);
  	}
  
  	guc_dirty = false;
*************** InitializeGUCOptions(void)
*** 3338,3343 ****
--- 3234,3348 ----
  	}
  }
  
+ static void
+ initialize_option(struct config_generic *gconf)
+ {
+ 	gconf->status = 0;
+ 	gconf->reset_source = PGC_S_DEFAULT;
+ 	gconf->source = PGC_S_DEFAULT;
+ 	gconf->stack = NULL;
+ 	gconf->sourcefile = NULL;
+ 	gconf->sourceline = 0;
+ 
+ 	switch (gconf->vartype)
+ 	{
+ 		case PGC_BOOL:
+ 			{
+ 				struct config_bool *conf = (struct config_bool *) gconf;
+ 
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %d",
+ 							 conf->gen.name, (int) conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_INT:
+ 			{
+ 				struct config_int *conf = (struct config_int *) gconf;
+ 
+ 				Assert(conf->boot_val >= conf->min);
+ 				Assert(conf->boot_val <= conf->max);
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %d",
+ 							 conf->gen.name, conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_REAL:
+ 			{
+ 				struct config_real *conf = (struct config_real *) gconf;
+ 
+ 				Assert(conf->boot_val >= conf->min);
+ 				Assert(conf->boot_val <= conf->max);
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %g",
+ 							 conf->gen.name, conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_STRING:
+ 			{
+ 				struct config_string *conf = (struct config_string *) gconf;
+ 				char	   *str;
+ 
+ 				*conf->variable = NULL;
+ 				conf->reset_val = NULL;
+ 
+ 				if (conf->boot_val == NULL)
+ 				{
+ 					/* leave the value NULL, do not call assign hook */
+ 					break;
+ 				}
+ 
+ 				str = guc_strdup(FATAL, conf->boot_val);
+ 				conf->reset_val = str;
+ 
+ 				if (conf->assign_hook)
+ 				{
+ 					const char *newstr;
+ 
+ 					newstr = (*conf->assign_hook) (str, true,
+ 												   PGC_S_DEFAULT);
+ 					if (newstr == NULL)
+ 					{
+ 						elog(FATAL, "failed to initialize %s to \"%s\"",
+ 							 conf->gen.name, str);
+ 					}
+ 					else if (newstr != str)
+ 					{
+ 						free(str);
+ 
+ 						/*
+ 						 * See notes in set_config_option about casting
+ 						 */
+ 						str = (char *) newstr;
+ 						conf->reset_val = str;
+ 					}
+ 				}
+ 				*conf->variable = str;
+ 				break;
+ 			}
+ 		case PGC_ENUM:
+ 			{
+ 				struct config_enum *conf = (struct config_enum *) gconf;
+ 
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %s",
+ 							 conf->gen.name, 
+ 							 config_enum_lookup_by_value(conf, conf->boot_val));
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 	}
+ }
  
  /*
   * Select the configuration files and data directory to be used, and
*************** define_custom_variable(struct config_gen
*** 5639,5644 ****
--- 5644,5650 ----
  	if (res == NULL)
  	{
  		/* No placeholder to replace, so just add it */
+ 		initialize_option(variable);
  		add_guc_variable(variable, ERROR);
  		return;
  	}
*************** define_custom_variable(struct config_gen
*** 5673,5678 ****
--- 5679,5686 ----
  		set_config_option(name, value,
  						  pHolder->gen.context, pHolder->gen.source,
  						  GUC_ACTION_SET, true);
+ 	else
+ 		initialize_option(variable);
  
  	/*
  	 * Free up as much as we conveniently can of the placeholder structure
*************** DefineCustomEnumVariable(const char *nam
*** 5804,5809 ****
--- 5812,5840 ----
  	var->assign_hook = assign_hook;
  	var->show_hook = show_hook;
  	define_custom_variable(&var->gen);
+ }
+ 
+ static const int config_varsize[] =
+ {
+ 	sizeof(struct config_bool),
+ 	sizeof(struct config_int),
+ 	sizeof(struct config_real),
+ 	sizeof(struct config_string),
+ 	sizeof(struct config_enum),
+ };
+ 
+ void
+ DefineCustomVariable(enum config_type type, const void *variable)
+ {
+ 	int		size = config_varsize[type];
+ 	const struct config_generic	   *var = variable;
+ 	struct config_generic		   *gen;
+ 
+ 	gen = (struct config_generic *) guc_malloc(ERROR, size);
+ 	memcpy(gen, var, size);
+ 	gen->name = guc_strdup(ERROR, var->name);
+ 	gen->vartype = type;
+ 	define_custom_variable(gen);
  }
  
  void
diff -cpr HEAD/src/include/commands/explain.h auto_explain/src/include/commands/explain.h
*** HEAD/src/include/commands/explain.h	Wed Jan  2 04:45:57 2008
--- auto_explain/src/include/commands/explain.h	Thu Oct  9 18:10:23 2008
*************** extern void ExplainOneUtility(Node *util
*** 41,44 ****
--- 41,47 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+ 							 bool analyze, bool verbose);
+ 
  #endif   /* EXPLAIN_H */
diff -cpr HEAD/src/include/executor/execdesc.h auto_explain/src/include/executor/execdesc.h
*** HEAD/src/include/executor/execdesc.h	Wed Jan  2 04:45:57 2008
--- auto_explain/src/include/executor/execdesc.h	Thu Oct  9 18:10:23 2008
***************
*** 20,25 ****
--- 20,27 ----
  #include "tcop/dest.h"
  
  
+ extern PGDLLIMPORT bool force_instrument;
+ 
  /* ----------------
   *		query descriptor:
   *
diff -cpr HEAD/src/include/utils/guc_tables.h auto_explain/src/include/utils/guc_tables.h
*** HEAD/src/include/utils/guc_tables.h	Tue Sep 30 19:52:14 2008
--- auto_explain/src/include/utils/guc_tables.h	Thu Oct  9 18:10:23 2008
*************** extern const char *config_enum_lookup_by
*** 242,246 ****
--- 242,247 ----
  extern bool config_enum_lookup_by_name(struct config_enum *record,
  									  const char *value, int *retval);
  
+ extern void DefineCustomVariable(enum config_type type, const void *variable);
  
  #endif   /* GUC_TABLES_H */
auto_explain.tgzapplication/octet-stream; name=auto_explain.tgzDownload
autoexplain.sgmlapplication/octet-stream; name=autoexplain.sgmlDownload
#38Alex Hunsaker
badalex@gmail.com
In reply to: ITAGAKI Takahiro (#37)
Re: auto_explain contrib moudle

On Thu, Oct 9, 2008 at 03:06, ITAGAKI Takahiro
<itagaki.takahiro@oss.ntt.co.jp> wrote:

Thanks for your reviewing, Alex.
I applied your comments to my patch.

Sorry for the late reply! Somehow I missed this, saw it on the commit
fest wiki :)

*custom_guc_flags-0828.patch
My only other concern is the changes to DefineCustom*() to tag the new
flags param. Now I think anyone who uses Custom gucs will want/should
be able to set that. I did not see any people in contrib using it but
did not look on PGfoundry. Do we need to document the change
somewhere for people who might be using it???

Now it is done with DefineCustomVariable(type, variable) and keep
existing functions as-is for backward compatibility.

Ok that seems better...

Some people will be happy if the functions are documented,
but we need to define 'stable-internal-functions' between
SPI (stable expoted functions) and unstable internal functions.

Right, thats why I was asking :)

*auto_explalin.c:
init_instrument()
The only "cleaner" way I can
see is to add a hook for CreateQueryDesc so we can overload
doInstrument and ExecInitNode will InstrAlloc them all for us.

I wanted to avoid modifying core codes as far as possible,
but I see it was ugly. Now I added 'force_instrument' global
variable as a hook for CreateQueryDesc.

Yeah, well if we are not to worried about it getting out of sync when
people add new node/scan types what you had before was probably ok. I
was just trying to stimulate my own and maybe others brains who are on
the list that might have better ideas. But at least now the commiter
has 2 options here :)

the only other comment I have is suset_assign() do we really need to
be a superuser if its all going to LOG ? There was some concern about
explaining security definer functions right? but surely a regular
explain on those shows the same thing as this explain? Or what am I
missing?

Almost logging options in postgres are only for superusers. So I think
auto_explain options should not be modified by non-superusers, too.

Ok thanks that makes sense.

#39Jeff Davis
pgsql@j-davis.com
In reply to: ITAGAKI Takahiro (#37)
Re: auto_explain contrib moudle

On Thu, 2008-10-09 at 19:06 +0900, ITAGAKI Takahiro wrote:

Thanks for your reviewing, Alex.
I applied your comments to my patch.

Hi,

This seems like a very useful feature, and it works nicely.

Initial comments:

1. Please sync with HEAD. There was a change made on Oct. 31st that
affects this patch, and prevents it from building properly.

2. Add it to the list of contrib modules to build in contrib/Makefile

Regards,
Jeff Davis

#40Jeff Davis
pgsql@j-davis.com
In reply to: ITAGAKI Takahiro (#37)
1 attachment(s)
Re: auto_explain contrib moudle

On Thu, 2008-10-09 at 19:06 +0900, ITAGAKI Takahiro wrote:

Thanks for your reviewing, Alex.
I applied your comments to my patch.

I made a few changes:

1. fixed some minor issues with auto_explain.sgml so that it would build
(and renamed to auto-explain.sgml to match other files)

2. added link in contrib.sgml

3. added line to contrib/Makefile to build the module

4. bundled it all up as one patch to make it easier to move around

It still needs to be merged with HEAD.

Regards,
Jeff Davis

Attachments:

auto-explain.1106.patchtext/x-patch; charset=utf-8; name=auto-explain.1106.patchDownload
diff --git a/contrib/Makefile b/contrib/Makefile
index 30f75c7..b25af3c 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 WANTED_DIRS = \
 		adminpack	\
+		auto_explain	\
 		btree_gist	\
 		chkpass		\
 		citext		\
diff --git a/contrib/auto_explain/Makefile b/contrib/auto_explain/Makefile
new file mode 100644
index 0000000..0d0ccc2
--- /dev/null
+++ b/contrib/auto_explain/Makefile
@@ -0,0 +1,13 @@
+MODULE_big = auto_explain
+OBJS = auto_explain.o
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/auto_explain
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
new file mode 100644
index 0000000..9bbff01
--- /dev/null
+++ b/contrib/auto_explain/auto_explain.c
@@ -0,0 +1,194 @@
+/*
+ * auto_explain.c
+ */
+#include "postgres.h"
+
+#include "commands/explain.h"
+#include "executor/instrument.h"
+#include "miscadmin.h"
+#include "utils/guc_tables.h"
+
+PG_MODULE_MAGIC;
+
+#define GUCNAME(name)		("explain." name)
+
+static int		explain_log_min_duration = -1;	/* msec or -1 */
+static bool		explain_log_analyze = false;
+static bool		explain_log_verbose = false;
+
+static bool		toplevel = true;
+static ExecutorRun_hook_type	prev_ExecutorRun = NULL;
+
+void	_PG_init(void);
+void	_PG_fini(void);
+
+static TupleTableSlot *explain_ExecutorRun(QueryDesc *queryDesc,
+										   ScanDirection direction,
+										   long count);
+static bool assign_log_min_duration(int newval, bool doit, GucSource source);
+static bool assign_log_analyze(bool newval, bool doit, GucSource source);
+static bool assign_log_verbose(bool newval, bool doit, GucSource source);
+
+static struct config_int def_log_min_duration =
+{
+	{
+		GUCNAME("log_min_duration"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Sets the minimum execution time above which plans will be logged.",
+		"Zero prints all plans. -1 turns this feature off.",
+		GUC_UNIT_MS
+	},
+	&explain_log_min_duration,
+	-1, -1, INT_MAX / 1000, assign_log_min_duration, NULL
+};
+
+static struct config_bool def_log_analyze =
+{
+	{
+		GUCNAME("log_analyze"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN ANALYZE for plan logging."
+	},
+	&explain_log_analyze,
+	false, assign_log_analyze, NULL
+};
+
+static struct config_bool def_log_verbose =
+{
+	{
+		GUCNAME("log_verbose"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN VERBOSE for plan logging."
+	},
+	&explain_log_verbose,
+	false, assign_log_verbose, NULL
+};
+
+void
+_PG_init(void)
+{
+	DefineCustomVariable(PGC_INT, &def_log_min_duration);
+	DefineCustomVariable(PGC_BOOL, &def_log_analyze);
+	DefineCustomVariable(PGC_BOOL, &def_log_verbose);
+
+	/* install ExecutorRun_hook */
+	prev_ExecutorRun = ExecutorRun_hook;
+	ExecutorRun_hook = explain_ExecutorRun;
+}
+
+void
+_PG_fini(void)
+{
+	/* uninstall ExecutorRun_hook */
+	ExecutorRun_hook = prev_ExecutorRun;
+}
+
+TupleTableSlot *
+explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
+{
+	TupleTableSlot *result;
+	
+	if (toplevel && explain_log_min_duration >= 0)
+	{
+		instr_time		starttime;
+		instr_time		duration;
+		double			msec;
+
+		/* Disable our hooks temporarily during the top-level query. */
+		toplevel = false;
+		PG_TRY();
+		{
+			INSTR_TIME_SET_CURRENT(starttime);
+
+			if (prev_ExecutorRun)
+				result = prev_ExecutorRun(queryDesc, direction, count);
+			else
+				result = standard_ExecutorRun(queryDesc, direction, count);
+
+			INSTR_TIME_SET_CURRENT(duration);
+			INSTR_TIME_SUBTRACT(duration, starttime);
+			msec = INSTR_TIME_GET_MILLISEC(duration);
+
+			/* Log plan if duration is exceeded. */
+			if (msec > explain_log_min_duration)
+			{
+				StringInfoData	buf;
+
+				initStringInfo(&buf);
+				ExplainOneResult(&buf, queryDesc,
+					queryDesc->doInstrument && explain_log_analyze,
+					explain_log_verbose);
+
+				/* Remove last line break */
+				if (buf.len > 0 && buf.data[buf.len - 1] == '\n')
+				{
+					buf.data[buf.len - 1] = '\0';
+					buf.len--;
+				}
+				ereport(LOG,
+						(errmsg("duration: %.3f ms  plan:\n%s",
+								msec, buf.data)));
+
+				pfree(buf.data);
+			}
+
+			toplevel = true;
+		}
+		PG_CATCH();
+		{
+			toplevel = true;
+			PG_RE_THROW();
+		}
+		PG_END_TRY();
+	}
+	else
+	{
+		/* ignore recursive executions, that are typically function calls */
+		if (prev_ExecutorRun)
+			result = prev_ExecutorRun(queryDesc, direction, count);
+		else
+			result = standard_ExecutorRun(queryDesc, direction, count);
+	}
+
+	return result;
+}
+
+/* Emulate PGC_SUSET for custom variables. */
+static bool
+suset_assign(GucSource source, const char *name)
+{
+	if (source >= PGC_S_CLIENT && !superuser())
+	{
+		ereport(GUC_complaint_elevel(source),
+				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+				 errmsg("permission denied to set parameter \"%s\"", name)));
+		return false;
+	}
+
+	return true;
+}
+
+static bool
+assign_log_min_duration(int newval, bool doit, GucSource source)
+{
+	if (!suset_assign(source, GUCNAME("log_min_duration")))
+		return false;
+	if (doit)
+		force_instrument = (newval >= 0);
+	return true;
+}
+
+static bool
+assign_log_analyze(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_analyze"));
+}
+
+static bool
+assign_log_verbose(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_verbose"));
+}
diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml
new file mode 100644
index 0000000..ea8da3b
--- /dev/null
+++ b/doc/src/sgml/auto-explain.sgml
@@ -0,0 +1,136 @@
+<sect1 id="autoexplain">
+ <title>auto_explain</title>
+
+ <indexterm zone="autoexplain">
+  <primary>auto_explain</primary>
+ </indexterm>
+
+ <para>
+  The <filename>auto_explain</filename> module provides a means for
+  logging execution plans that execution times are longer than configuration.
+ </para>
+
+ <para>
+  You can <command>LOAD</> this module dynamically or preload it automatically with
+  <varname>shared_preload_libraries</> or <varname>local_preload_libraries</>.
+ </para>
+
+ <sect2>
+  <title>The <structname>auto_explain</structname> view</title>
+
+  <para>
+   The definitions of the columns exposed by the view are:
+  </para>
+
+  <para>
+   There is one row for each statement. Statements are grouped when they have
+   same SQL text, are in the same database, and are executed by the same user.
+  </para>
+
+  <para>
+   Because of security restriction, non-super users cannot see query strings
+   executed by other users.
+  </para>
+ </sect2>
+
+ <sect2>
+  <title>Configuration parameters</title>
+
+  <variablelist>
+   <varlistentry>
+    <term>
+     <varname>explain.log_min_duration</varname> (<type>integer</type>)
+    </term>
+    <listitem>
+     <para>
+      <varname>explain.log_min_duration</varname> is the minimum execution time
+      in milliseconds which execution plan will be logged. Setting this to zero
+      prints all plans. Minus-one (the default) disables logging plans.
+      For example, if you set it to <literal>250ms</literal> then all plan
+      that run 250ms or longer in executor will be logged.
+      Enabling this parameter can be helpful in tracking down unoptimized queries
+      in your applications. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_analyze</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_analyze</varname> enables to use EXPLAIN ANALYZE when
+      an execution plan is logged by <varname>explain.log_min_duration</varname>.
+      This parameter is off by default. Only superusers can change this setting.
+     </para>
+     <para>
+      NOTE: If you set the parameter on, instrument timers are enabled even if
+      you don't use EXPLAIN ANALYZE. This has some overhead to execute plans.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_verbose</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_verbose</varname> enables to use EXPLAIN VERBOSE when
+      an execution plan is logged by <varname>explain.log_min_duration</varname>.
+      This parameter is off by default. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+
+  <para>
+   If you set these explain.* parameters in your postgresql.conf,
+   you also need to add 'explain' in <varname>custom_variable_classes</>.
+  </para>
+
+  <programlisting>
+  # postgresql.conf
+  shared_preload_libraries = 'auto_explain'
+  
+  custom_variable_classes = 'explain'
+  explain.log_min_duration = 3s
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Sample output</title>
+
+  <programlisting>
+  postgres=# LOAD 'auto_explain';
+  postgres=# SET explain.log_min_duration = 0;
+  postgres=# SELECT count(*)
+               FROM pg_class, pg_index
+              WHERE oid = indrelid AND indisunique;
+  
+  LOG:  duration: 0.986 ms  plan:
+          Aggregate  (cost=14.90..14.91 rows=1 width=0)
+            ->  Hash Join  (cost=3.91..14.70 rows=81 width=0)
+                  Hash Cond: (pg_class.oid = pg_index.indrelid)
+                  ->  Seq Scan on pg_class  (cost=0.00..8.27 rows=227 width=4)
+                  ->  Hash  (cost=2.90..2.90 rows=81 width=4)
+                        ->  Seq Scan on pg_index  (cost=0.00..2.90 rows=81 width=4)
+                              Filter: indisunique
+  STATEMENT:  SELECT count(*)
+            FROM pg_class, pg_index
+           WHERE oid = indrelid AND indisunique;
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Authors</title>
+
+  <para>
+   Takahiro Itagaki <email>itagaki.takahiro@oss.ntt.co.jp</email>
+  </para>
+ </sect2>
+
+</sect1>
diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
index 72d8828..0108da3 100644
--- a/doc/src/sgml/contrib.sgml
+++ b/doc/src/sgml/contrib.sgml
@@ -79,6 +79,7 @@ psql -d dbname -f <replaceable>SHAREDIR</>/contrib/<replaceable>module</>.sql
  </para>
 
  &adminpack;
+ &auto-explain;
  &btree-gist;
  &chkpass;
  &citext;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index b0538b6..68db2b9 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -92,6 +92,7 @@
 <!-- contrib information -->
 <!entity contrib         SYSTEM "contrib.sgml">
 <!entity adminpack       SYSTEM "adminpack.sgml">
+<!entity auto-explain    SYSTEM "auto-explain.sgml">
 <!entity btree-gist      SYSTEM "btree-gist.sgml">
 <!entity chkpass         SYSTEM "chkpass.sgml">
 <!entity citext          SYSTEM "citext.sgml">
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c0c53cf..4d46a34 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -224,7 +224,6 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	QueryDesc  *queryDesc;
 	instr_time	starttime;
 	double		totaltime = 0;
-	ExplainState *es;
 	StringInfoData buf;
 	int			eflags;
 
@@ -265,17 +264,8 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 		totaltime += elapsed_time(&starttime);
 	}
 
-	es = (ExplainState *) palloc0(sizeof(ExplainState));
-
-	es->printTList = stmt->verbose;
-	es->printAnalyze = stmt->analyze;
-	es->pstmt = queryDesc->plannedstmt;
-	es->rtable = queryDesc->plannedstmt->rtable;
-
 	initStringInfo(&buf);
-	explain_outNode(&buf,
-					queryDesc->plannedstmt->planTree, queryDesc->planstate,
-					NULL, 0, es);
+	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
 
 	/*
 	 * If we ran the command, run any AFTER triggers it queued.  (Note this
@@ -290,7 +280,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	}
 
 	/* Print info about runtime of triggers */
-	if (es->printAnalyze)
+	if (stmt->analyze)
 	{
 		ResultRelInfo *rInfo;
 		bool		show_relname;
@@ -335,7 +325,26 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	do_text_output_multiline(tstate, buf.data);
 
 	pfree(buf.data);
-	pfree(es);
+}
+
+/*
+ * ExplainOneResult -
+ *	  converts a Plan node into ascii string and appends it to 'str'
+ */
+void
+ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+				 bool analyze, bool verbose)
+{
+	ExplainState	es = { 0 };
+
+	es.printTList = verbose;
+	es.printAnalyze = analyze;
+	es.pstmt = queryDesc->plannedstmt;
+	es.rtable = queryDesc->plannedstmt->rtable;
+
+	explain_outNode(str,
+					queryDesc->plannedstmt->planTree, queryDesc->planstate,
+					NULL, 0, &es);
 }
 
 /*
diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c
index a933d3c..ff2df9b 100644
--- a/src/backend/tcop/pquery.c
+++ b/src/backend/tcop/pquery.c
@@ -32,7 +32,7 @@
  * if there are several).
  */
 Portal		ActivePortal = NULL;
-
+bool		force_instrument = false;
 
 static void ProcessQuery(PlannedStmt *plan,
 			 ParamListInfo params,
@@ -76,7 +76,7 @@ CreateQueryDesc(PlannedStmt *plannedstmt,
 	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
 	qd->dest = dest;			/* output dest */
 	qd->params = params;		/* parameter values passed into query */
-	qd->doInstrument = doInstrument;	/* instrumentation wanted? */
+	qd->doInstrument = force_instrument || doInstrument;	/* instrumentation wanted? */
 
 	/* null these fields until set by ExecutorStart */
 	qd->tupDesc = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 58851e5..015f06d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -169,6 +169,7 @@ static const char *assign_pgstat_temp_directory(const char *newval, bool doit, G
 
 static char *config_enum_get_options(struct config_enum *record, 
 									 const char *prefix, const char *suffix);
+static void initialize_option(struct config_generic *gconf);
 
 
 /*
@@ -3177,112 +3178,7 @@ InitializeGUCOptions(void)
 	for (i = 0; i < num_guc_variables; i++)
 	{
 		struct config_generic *gconf = guc_variables[i];
-
-		gconf->status = 0;
-		gconf->reset_source = PGC_S_DEFAULT;
-		gconf->source = PGC_S_DEFAULT;
-		gconf->stack = NULL;
-		gconf->sourcefile = NULL;
-		gconf->sourceline = 0;
-
-		switch (gconf->vartype)
-		{
-			case PGC_BOOL:
-				{
-					struct config_bool *conf = (struct config_bool *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, (int) conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_INT:
-				{
-					struct config_int *conf = (struct config_int *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_REAL:
-				{
-					struct config_real *conf = (struct config_real *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %g",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_STRING:
-				{
-					struct config_string *conf = (struct config_string *) gconf;
-					char	   *str;
-
-					*conf->variable = NULL;
-					conf->reset_val = NULL;
-
-					if (conf->boot_val == NULL)
-					{
-						/* leave the value NULL, do not call assign hook */
-						break;
-					}
-
-					str = guc_strdup(FATAL, conf->boot_val);
-					conf->reset_val = str;
-
-					if (conf->assign_hook)
-					{
-						const char *newstr;
-
-						newstr = (*conf->assign_hook) (str, true,
-													   PGC_S_DEFAULT);
-						if (newstr == NULL)
-						{
-							elog(FATAL, "failed to initialize %s to \"%s\"",
-								 conf->gen.name, str);
-						}
-						else if (newstr != str)
-						{
-							free(str);
-
-							/*
-							 * See notes in set_config_option about casting
-							 */
-							str = (char *) newstr;
-							conf->reset_val = str;
-						}
-					}
-					*conf->variable = str;
-					break;
-				}
-			case PGC_ENUM:
-				{
-					struct config_enum *conf = (struct config_enum *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %s",
-								 conf->gen.name, 
-								 config_enum_lookup_by_value(conf, conf->boot_val));
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-		}
+		initialize_option(gconf);
 	}
 
 	guc_dirty = false;
@@ -3338,6 +3234,115 @@ InitializeGUCOptions(void)
 	}
 }
 
+static void
+initialize_option(struct config_generic *gconf)
+{
+	gconf->status = 0;
+	gconf->reset_source = PGC_S_DEFAULT;
+	gconf->source = PGC_S_DEFAULT;
+	gconf->stack = NULL;
+	gconf->sourcefile = NULL;
+	gconf->sourceline = 0;
+
+	switch (gconf->vartype)
+	{
+		case PGC_BOOL:
+			{
+				struct config_bool *conf = (struct config_bool *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, (int) conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_INT:
+			{
+				struct config_int *conf = (struct config_int *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_REAL:
+			{
+				struct config_real *conf = (struct config_real *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %g",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_STRING:
+			{
+				struct config_string *conf = (struct config_string *) gconf;
+				char	   *str;
+
+				*conf->variable = NULL;
+				conf->reset_val = NULL;
+
+				if (conf->boot_val == NULL)
+				{
+					/* leave the value NULL, do not call assign hook */
+					break;
+				}
+
+				str = guc_strdup(FATAL, conf->boot_val);
+				conf->reset_val = str;
+
+				if (conf->assign_hook)
+				{
+					const char *newstr;
+
+					newstr = (*conf->assign_hook) (str, true,
+												   PGC_S_DEFAULT);
+					if (newstr == NULL)
+					{
+						elog(FATAL, "failed to initialize %s to \"%s\"",
+							 conf->gen.name, str);
+					}
+					else if (newstr != str)
+					{
+						free(str);
+
+						/*
+						 * See notes in set_config_option about casting
+						 */
+						str = (char *) newstr;
+						conf->reset_val = str;
+					}
+				}
+				*conf->variable = str;
+				break;
+			}
+		case PGC_ENUM:
+			{
+				struct config_enum *conf = (struct config_enum *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %s",
+							 conf->gen.name, 
+							 config_enum_lookup_by_value(conf, conf->boot_val));
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+	}
+}
 
 /*
  * Select the configuration files and data directory to be used, and
@@ -5639,6 +5644,7 @@ define_custom_variable(struct config_generic * variable)
 	if (res == NULL)
 	{
 		/* No placeholder to replace, so just add it */
+		initialize_option(variable);
 		add_guc_variable(variable, ERROR);
 		return;
 	}
@@ -5673,6 +5679,8 @@ define_custom_variable(struct config_generic * variable)
 		set_config_option(name, value,
 						  pHolder->gen.context, pHolder->gen.source,
 						  GUC_ACTION_SET, true);
+	else
+		initialize_option(variable);
 
 	/*
 	 * Free up as much as we conveniently can of the placeholder structure
@@ -5806,6 +5814,29 @@ DefineCustomEnumVariable(const char *name,
 	define_custom_variable(&var->gen);
 }
 
+static const int config_varsize[] =
+{
+	sizeof(struct config_bool),
+	sizeof(struct config_int),
+	sizeof(struct config_real),
+	sizeof(struct config_string),
+	sizeof(struct config_enum),
+};
+
+void
+DefineCustomVariable(enum config_type type, const void *variable)
+{
+	int		size = config_varsize[type];
+	const struct config_generic	   *var = variable;
+	struct config_generic		   *gen;
+
+	gen = (struct config_generic *) guc_malloc(ERROR, size);
+	memcpy(gen, var, size);
+	gen->name = guc_strdup(ERROR, var->name);
+	gen->vartype = type;
+	define_custom_variable(gen);
+}
+
 void
 EmitWarningsOnPlaceholders(const char *className)
 {
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 5b808d5..cf07915 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -41,4 +41,7 @@ extern void ExplainOneUtility(Node *utilityStmt, ExplainStmt *stmt,
 extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 			   ExplainStmt *stmt, TupOutputState *tstate);
 
+extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+							 bool analyze, bool verbose);
+
 #endif   /* EXPLAIN_H */
diff --git a/src/include/executor/execdesc.h b/src/include/executor/execdesc.h
index 2e2c159..854721c 100644
--- a/src/include/executor/execdesc.h
+++ b/src/include/executor/execdesc.h
@@ -20,6 +20,8 @@
 #include "tcop/dest.h"
 
 
+extern PGDLLIMPORT bool force_instrument;
+
 /* ----------------
  *		query descriptor:
  *
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index 8791660..3982a27 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -242,5 +242,6 @@ extern const char *config_enum_lookup_by_value(struct config_enum *record, int v
 extern bool config_enum_lookup_by_name(struct config_enum *record,
 									  const char *value, int *retval);
 
+extern void DefineCustomVariable(enum config_type type, const void *variable);
 
 #endif   /* GUC_TABLES_H */
#41Martin Pihlak
martin.pihlak@gmail.com
In reply to: Jeff Davis (#40)
1 attachment(s)
Re: auto_explain contrib moudle

Jeff Davis wrote:

It still needs to be merged with HEAD.

ExecutorRun function signature has changed to return void. Other than that
it seems OK. I'll add a few additional notes:

One thing that I noticed was that tab completion queries are also explained
if "explain.log_min_duration" is set to zero. Actually this also applies to
psql \dX commands. Don't know if this is deliberate or not. Example:

load 'auto_explain';
set explain.log_min_duration = 0;
set client_min_messages = LOG;
select * from pg_catalog.^ILOG: duration: 4.713 ms plan:
Limit (cost=27.41..27.44 rows=3 width=85) (actual time=4.023..4.418 rows=75 loops=1)
...

I have a feeling that this can become somewhat annoying during an interactive
troubleshooting session.

Another thing is a feature I am interested in -- ability to auto-explain statements
execututed from within functions. I'm thinking of adding an extra boolean GUC --
"explain.log_nested_statements" (defaults to false). Quick test seems to give
expected results, but there maybe something I missed.

regards,
Martin

Attachments:

explain-nested.patchtext/x-diff; name=explain-nested.patchDownload
*** a/contrib/auto_explain/auto_explain.c
--- b/contrib/auto_explain/auto_explain.c
***************
*** 15,20 **** PG_MODULE_MAGIC;
--- 15,21 ----
  static int		explain_log_min_duration = -1;	/* msec or -1 */
  static bool		explain_log_analyze = false;
  static bool		explain_log_verbose = false;
+ static bool		explain_log_nested = false;
  
  static bool		toplevel = true;
  static ExecutorRun_hook_type	prev_ExecutorRun = NULL;
***************
*** 28,33 **** static void explain_ExecutorRun(QueryDesc *queryDesc,
--- 29,35 ----
  static bool assign_log_min_duration(int newval, bool doit, GucSource source);
  static bool assign_log_analyze(bool newval, bool doit, GucSource source);
  static bool assign_log_verbose(bool newval, bool doit, GucSource source);
+ static bool assign_log_nested(bool newval, bool doit, GucSource source);
  
  static struct config_int def_log_min_duration =
  {
***************
*** 67,78 **** static struct config_bool def_log_verbose =
--- 69,93 ----
  	false, assign_log_verbose, NULL
  };
  
+ static struct config_bool def_log_nested_statements =
+ {
+ 	{
+ 		GUCNAME("log_nested_statements"),
+ 		PGC_USERSET,
+ 		STATS_MONITORING,
+ 		"Log nested statements."
+ 	},
+ 	&explain_log_nested,
+ 	false, assign_log_nested, NULL
+ };
+ 
  void
  _PG_init(void)
  {
  	DefineCustomVariable(PGC_INT, &def_log_min_duration);
  	DefineCustomVariable(PGC_BOOL, &def_log_analyze);
  	DefineCustomVariable(PGC_BOOL, &def_log_verbose);
+ 	DefineCustomVariable(PGC_BOOL, &def_log_nested_statements);
  
  	/* install ExecutorRun_hook */
  	prev_ExecutorRun = ExecutorRun_hook;
***************
*** 89,95 **** _PG_fini(void)
  void
  explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
  {
! 	if (toplevel && explain_log_min_duration >= 0)
  	{
  		instr_time		starttime;
  		instr_time		duration;
--- 104,110 ----
  void
  explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
  {
! 	if ((toplevel || explain_log_nested) && explain_log_min_duration >= 0)
  	{
  		instr_time		starttime;
  		instr_time		duration;
***************
*** 188,190 **** assign_log_verbose(bool newval, bool doit, GucSource source)
--- 203,211 ----
  {
  	return suset_assign(source, GUCNAME("log_verbose"));
  }
+ 
+ static bool
+ assign_log_nested(bool newval, bool doit, GucSource source)
+ {
+ 	return suset_assign(source, GUCNAME("log_nested_statements"));
+ }
#42Jeff Davis
pgsql@j-davis.com
In reply to: Martin Pihlak (#41)
Re: auto_explain contrib moudle

On Fri, 2008-11-07 at 18:03 +0200, Martin Pihlak wrote:

One thing that I noticed was that tab completion queries are also explained
if "explain.log_min_duration" is set to zero. Actually this also applies to
psql \dX commands. Don't know if this is deliberate or not. Example:

It's logged at the LOG level, just like log_min_duration_statement, and
seems to have the same behavior. What do you think it should do
differently?

Another thing is a feature I am interested in -- ability to auto-explain statements
execututed from within functions. I'm thinking of adding an extra boolean GUC --
"explain.log_nested_statements" (defaults to false). Quick test seems to give
expected results, but there maybe something I missed.

I like that idea, I'll take a look at it.

Regards,
Jeff Davis

#43Martin Pihlak
martin.pihlak@gmail.com
In reply to: Jeff Davis (#42)
Re: auto_explain contrib moudle

Jeff Davis wrote:

It's logged at the LOG level, just like log_min_duration_statement, and
seems to have the same behavior. What do you think it should do
differently?

There was actually a patch to disable the psql notices, but there were
some concerns and I think it was removed:
http://archives.postgresql.org/pgsql-hackers/2008-07/msg01264.php
http://archives.postgresql.org/pgsql-hackers/2008-09/msg01752.php

Patch is at:
http://archives.postgresql.org/pgsql-hackers/2008-08/msg01274.php
and seems to get rid of the annoying messages. If there aren't any
major issues with it, I think it should be re-added.

regards,
Martin

#44Jeff Davis
pgsql@j-davis.com
In reply to: Martin Pihlak (#43)
Re: auto_explain contrib moudle

On Fri, 2008-11-07 at 22:23 +0200, Martin Pihlak wrote:

Patch is at:
http://archives.postgresql.org/pgsql-hackers/2008-08/msg01274.php
and seems to get rid of the annoying messages. If there aren't any
major issues with it, I think it should be re-added.

I still don't understand why this psql patch is desirable. Who sets
their client_min_messages to LOG in psql? And if they do, why would they
expect different behavior that they always got from the already-existing
GUC log_min_duration_statement?

Regards,
Jeff Davis

#45Tom Lane
tgl@sss.pgh.pa.us
In reply to: Martin Pihlak (#43)
Re: auto_explain contrib moudle

Martin Pihlak <martin.pihlak@gmail.com> writes:

There was actually a patch to disable the psql notices, but there were
some concerns and I think it was removed:
http://archives.postgresql.org/pgsql-hackers/2008-07/msg01264.php
http://archives.postgresql.org/pgsql-hackers/2008-09/msg01752.php

Patch is at:
http://archives.postgresql.org/pgsql-hackers/2008-08/msg01274.php
and seems to get rid of the annoying messages. If there aren't any
major issues with it, I think it should be re-added.

As I said in the first-quoted message, that patch was simply a band-aid
trying to cover poorly designed behavior; and it still is. The fact is
that most clients will not react very gracefully to unexpected notice
messages, especially not large numbers of same.

regards, tom lane

#46Martin Pihlak
martin.pihlak@gmail.com
In reply to: Jeff Davis (#44)
Re: auto_explain contrib moudle

Jeff Davis wrote:

I still don't understand why this psql patch is desirable. Who sets
their client_min_messages to LOG in psql? And if they do, why would they
expect different behavior that they always got from the already-existing
GUC log_min_duration_statement?

I know a few ;) In my environment the auto-explain is especially useful when
used from within psql. Server logs are not always easy to get to, and it is
difficult to extract the interesting bits (large files and lots of log traffic).

For me the primary use of auto-explain would be interactive troubleshooting.
The troublesome statements usually involve several nested function calls and
are tedious to trace manually. With auto-explain I fire up psql, load the
module, set the client log level, run the statements and immediately see
what's going on. I bet that lot of the developers and QA folk would use it
similarly.

You are of course right about the log_min_duration_statement, also the
log_executor_stats etc. behave similarly. So indeed, the "ignore notices" patch
is not necessarily part of auto-explain.

Regards,
Martin

#47Jeff Davis
pgsql@j-davis.com
In reply to: Martin Pihlak (#46)
Re: auto_explain contrib moudle

On Sat, 2008-11-08 at 12:18 +0200, Martin Pihlak wrote:

For me the primary use of auto-explain would be interactive troubleshooting.
The troublesome statements usually involve several nested function calls and
are tedious to trace manually. With auto-explain I fire up psql, load the
module, set the client log level, run the statements and immediately see
what's going on. I bet that lot of the developers and QA folk would use it
similarly.

I think the cost in terms of inconsistency here is just too high for the
benefit. If you set the client_min_messages to LOG, you should really
get *all* the log messages, not all except for those that happen to
occur when psql is in some implicit mode that's invisible to the user.

By the way, a possible solution for the problem you describe is to
simply set explain.log_min_duration=1s or something, so that you won't
see tab completion queries (or, if you do, this is the least of your
problems). Or, just disable tab completion.

You are of course right about the log_min_duration_statement, also the
log_executor_stats etc. behave similarly. So indeed, the "ignore notices" patch
is not necessarily part of auto-explain.

I agree that there might be room for improvement in psql's handling of
notices, and that the logging system might be made more flexible. But
let's just keep it simple so that we get something in 8.4. I think
auto-explain will be very useful to a lot of people as-is.

Regards,
Jeff Davis

#48Jeff Davis
pgsql@j-davis.com
In reply to: Martin Pihlak (#41)
1 attachment(s)
Re: auto_explain contrib moudle

On Fri, 2008-11-07 at 18:03 +0200, Martin Pihlak wrote:

Another thing is a feature I am interested in -- ability to auto-explain statements
execututed from within functions. I'm thinking of adding an extra boolean GUC --
"explain.log_nested_statements" (defaults to false). Quick test seems to give
expected results, but there maybe something I missed.

Thanks.

I applied this patch for log_nested_statements and I merged with HEAD.
Current patch attached.

One thing I'm unsure of (this question is for ITAGAKI Takahiro): why is
it necessary to define a new function DefineCustomVariable(), when there
are already functions DefineCustomBoolVariable() and
DefineCustomIntVariable()?

Regards,
Jeff Davis

Attachments:

auto-explain.1108.patchtext/x-patch; charset=utf-8; name=auto-explain.1108.patchDownload
diff --git a/contrib/Makefile b/contrib/Makefile
index 30f75c7..b25af3c 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 WANTED_DIRS = \
 		adminpack	\
+		auto_explain	\
 		btree_gist	\
 		chkpass		\
 		citext		\
diff --git a/contrib/auto_explain/Makefile b/contrib/auto_explain/Makefile
new file mode 100644
index 0000000..0d0ccc2
--- /dev/null
+++ b/contrib/auto_explain/Makefile
@@ -0,0 +1,13 @@
+MODULE_big = auto_explain
+OBJS = auto_explain.o
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/auto_explain
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
new file mode 100644
index 0000000..5f582c3
--- /dev/null
+++ b/contrib/auto_explain/auto_explain.c
@@ -0,0 +1,213 @@
+/*
+ * auto_explain.c
+ */
+#include "postgres.h"
+
+#include "commands/explain.h"
+#include "executor/instrument.h"
+#include "miscadmin.h"
+#include "utils/guc_tables.h"
+
+PG_MODULE_MAGIC;
+
+#define GUCNAME(name)		("explain." name)
+
+static int		explain_log_min_duration = -1;	/* msec or -1 */
+static bool		explain_log_analyze = false;
+static bool		explain_log_verbose = false;
+static bool		explain_log_nested = false;
+
+static bool		toplevel = true;
+static ExecutorRun_hook_type	prev_ExecutorRun = NULL;
+
+void	_PG_init(void);
+void	_PG_fini(void);
+
+static void explain_ExecutorRun(QueryDesc *queryDesc,
+										   ScanDirection direction,
+										   long count);
+static bool assign_log_min_duration(int newval, bool doit, GucSource source);
+static bool assign_log_analyze(bool newval, bool doit, GucSource source);
+static bool assign_log_verbose(bool newval, bool doit, GucSource source);
+static bool assign_log_nested(bool newval, bool doit, GucSource source);
+
+static struct config_int def_log_min_duration =
+{
+	{
+		GUCNAME("log_min_duration"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Sets the minimum execution time above which plans will be logged.",
+		"Zero prints all plans. -1 turns this feature off.",
+		GUC_UNIT_MS
+	},
+	&explain_log_min_duration,
+	-1, -1, INT_MAX / 1000, assign_log_min_duration, NULL
+};
+
+static struct config_bool def_log_analyze =
+{
+	{
+		GUCNAME("log_analyze"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN ANALYZE for plan logging."
+	},
+	&explain_log_analyze,
+	false, assign_log_analyze, NULL
+};
+
+static struct config_bool def_log_verbose =
+{
+	{
+		GUCNAME("log_verbose"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN VERBOSE for plan logging."
+	},
+	&explain_log_verbose,
+	false, assign_log_verbose, NULL
+};
+
+static struct config_bool def_log_nested_statements =
+{
+	{
+		GUCNAME("log_nested_statements"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Log nested statements."
+	},
+	&explain_log_nested,
+	false, assign_log_nested, NULL
+};
+
+void
+_PG_init(void)
+{
+	DefineCustomVariable(PGC_INT, &def_log_min_duration);
+	DefineCustomVariable(PGC_BOOL, &def_log_analyze);
+	DefineCustomVariable(PGC_BOOL, &def_log_verbose);
+	DefineCustomVariable(PGC_BOOL, &def_log_nested_statements);
+
+	/* install ExecutorRun_hook */
+	prev_ExecutorRun = ExecutorRun_hook;
+	ExecutorRun_hook = explain_ExecutorRun;
+}
+
+void
+_PG_fini(void)
+{
+	/* uninstall ExecutorRun_hook */
+	ExecutorRun_hook = prev_ExecutorRun;
+}
+
+void
+explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
+{
+	if ((toplevel || explain_log_nested) && explain_log_min_duration >= 0)
+	{
+		instr_time		starttime;
+		instr_time		duration;
+		double			msec;
+
+		/* Disable our hooks temporarily during the top-level query. */
+		toplevel = false;
+		PG_TRY();
+		{
+			INSTR_TIME_SET_CURRENT(starttime);
+
+			if (prev_ExecutorRun)
+				prev_ExecutorRun(queryDesc, direction, count);
+			else
+				standard_ExecutorRun(queryDesc, direction, count);
+
+			INSTR_TIME_SET_CURRENT(duration);
+			INSTR_TIME_SUBTRACT(duration, starttime);
+			msec = INSTR_TIME_GET_MILLISEC(duration);
+
+			/* Log plan if duration is exceeded. */
+			if (msec > explain_log_min_duration)
+			{
+				StringInfoData	buf;
+
+				initStringInfo(&buf);
+				ExplainOneResult(&buf, queryDesc,
+					queryDesc->doInstrument && explain_log_analyze,
+					explain_log_verbose);
+
+				/* Remove last line break */
+				if (buf.len > 0 && buf.data[buf.len - 1] == '\n')
+				{
+					buf.data[buf.len - 1] = '\0';
+					buf.len--;
+				}
+				ereport(LOG,
+						(errmsg("duration: %.3f ms  plan:\n%s",
+								msec, buf.data)));
+
+				pfree(buf.data);
+			}
+
+			toplevel = true;
+		}
+		PG_CATCH();
+		{
+			toplevel = true;
+			PG_RE_THROW();
+		}
+		PG_END_TRY();
+	}
+	else
+	{
+		/* ignore recursive executions, that are typically function calls */
+		if (prev_ExecutorRun)
+			prev_ExecutorRun(queryDesc, direction, count);
+		else
+			standard_ExecutorRun(queryDesc, direction, count);
+	}
+
+	return;
+}
+
+/* Emulate PGC_SUSET for custom variables. */
+static bool
+suset_assign(GucSource source, const char *name)
+{
+	if (source >= PGC_S_CLIENT && !superuser())
+	{
+		ereport(GUC_complaint_elevel(source),
+				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+				 errmsg("permission denied to set parameter \"%s\"", name)));
+		return false;
+	}
+
+	return true;
+}
+
+static bool
+assign_log_min_duration(int newval, bool doit, GucSource source)
+{
+	if (!suset_assign(source, GUCNAME("log_min_duration")))
+		return false;
+	if (doit)
+		force_instrument = (newval >= 0);
+	return true;
+}
+
+static bool
+assign_log_analyze(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_analyze"));
+}
+
+static bool
+assign_log_verbose(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_verbose"));
+}
+
+static bool
+assign_log_nested(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_nested_statements"));
+}
diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml
new file mode 100644
index 0000000..ea8da3b
--- /dev/null
+++ b/doc/src/sgml/auto-explain.sgml
@@ -0,0 +1,136 @@
+<sect1 id="autoexplain">
+ <title>auto_explain</title>
+
+ <indexterm zone="autoexplain">
+  <primary>auto_explain</primary>
+ </indexterm>
+
+ <para>
+  The <filename>auto_explain</filename> module provides a means for
+  logging execution plans that execution times are longer than configuration.
+ </para>
+
+ <para>
+  You can <command>LOAD</> this module dynamically or preload it automatically with
+  <varname>shared_preload_libraries</> or <varname>local_preload_libraries</>.
+ </para>
+
+ <sect2>
+  <title>The <structname>auto_explain</structname> view</title>
+
+  <para>
+   The definitions of the columns exposed by the view are:
+  </para>
+
+  <para>
+   There is one row for each statement. Statements are grouped when they have
+   same SQL text, are in the same database, and are executed by the same user.
+  </para>
+
+  <para>
+   Because of security restriction, non-super users cannot see query strings
+   executed by other users.
+  </para>
+ </sect2>
+
+ <sect2>
+  <title>Configuration parameters</title>
+
+  <variablelist>
+   <varlistentry>
+    <term>
+     <varname>explain.log_min_duration</varname> (<type>integer</type>)
+    </term>
+    <listitem>
+     <para>
+      <varname>explain.log_min_duration</varname> is the minimum execution time
+      in milliseconds which execution plan will be logged. Setting this to zero
+      prints all plans. Minus-one (the default) disables logging plans.
+      For example, if you set it to <literal>250ms</literal> then all plan
+      that run 250ms or longer in executor will be logged.
+      Enabling this parameter can be helpful in tracking down unoptimized queries
+      in your applications. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_analyze</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_analyze</varname> enables to use EXPLAIN ANALYZE when
+      an execution plan is logged by <varname>explain.log_min_duration</varname>.
+      This parameter is off by default. Only superusers can change this setting.
+     </para>
+     <para>
+      NOTE: If you set the parameter on, instrument timers are enabled even if
+      you don't use EXPLAIN ANALYZE. This has some overhead to execute plans.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_verbose</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_verbose</varname> enables to use EXPLAIN VERBOSE when
+      an execution plan is logged by <varname>explain.log_min_duration</varname>.
+      This parameter is off by default. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+
+  <para>
+   If you set these explain.* parameters in your postgresql.conf,
+   you also need to add 'explain' in <varname>custom_variable_classes</>.
+  </para>
+
+  <programlisting>
+  # postgresql.conf
+  shared_preload_libraries = 'auto_explain'
+  
+  custom_variable_classes = 'explain'
+  explain.log_min_duration = 3s
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Sample output</title>
+
+  <programlisting>
+  postgres=# LOAD 'auto_explain';
+  postgres=# SET explain.log_min_duration = 0;
+  postgres=# SELECT count(*)
+               FROM pg_class, pg_index
+              WHERE oid = indrelid AND indisunique;
+  
+  LOG:  duration: 0.986 ms  plan:
+          Aggregate  (cost=14.90..14.91 rows=1 width=0)
+            ->  Hash Join  (cost=3.91..14.70 rows=81 width=0)
+                  Hash Cond: (pg_class.oid = pg_index.indrelid)
+                  ->  Seq Scan on pg_class  (cost=0.00..8.27 rows=227 width=4)
+                  ->  Hash  (cost=2.90..2.90 rows=81 width=4)
+                        ->  Seq Scan on pg_index  (cost=0.00..2.90 rows=81 width=4)
+                              Filter: indisunique
+  STATEMENT:  SELECT count(*)
+            FROM pg_class, pg_index
+           WHERE oid = indrelid AND indisunique;
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Authors</title>
+
+  <para>
+   Takahiro Itagaki <email>itagaki.takahiro@oss.ntt.co.jp</email>
+  </para>
+ </sect2>
+
+</sect1>
diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
index 72d8828..0108da3 100644
--- a/doc/src/sgml/contrib.sgml
+++ b/doc/src/sgml/contrib.sgml
@@ -79,6 +79,7 @@ psql -d dbname -f <replaceable>SHAREDIR</>/contrib/<replaceable>module</>.sql
  </para>
 
  &adminpack;
+ &auto-explain;
  &btree-gist;
  &chkpass;
  &citext;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index b0538b6..68db2b9 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -92,6 +92,7 @@
 <!-- contrib information -->
 <!entity contrib         SYSTEM "contrib.sgml">
 <!entity adminpack       SYSTEM "adminpack.sgml">
+<!entity auto-explain    SYSTEM "auto-explain.sgml">
 <!entity btree-gist      SYSTEM "btree-gist.sgml">
 <!entity chkpass         SYSTEM "chkpass.sgml">
 <!entity citext          SYSTEM "citext.sgml">
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c0c53cf..4d46a34 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -224,7 +224,6 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	QueryDesc  *queryDesc;
 	instr_time	starttime;
 	double		totaltime = 0;
-	ExplainState *es;
 	StringInfoData buf;
 	int			eflags;
 
@@ -265,17 +264,8 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 		totaltime += elapsed_time(&starttime);
 	}
 
-	es = (ExplainState *) palloc0(sizeof(ExplainState));
-
-	es->printTList = stmt->verbose;
-	es->printAnalyze = stmt->analyze;
-	es->pstmt = queryDesc->plannedstmt;
-	es->rtable = queryDesc->plannedstmt->rtable;
-
 	initStringInfo(&buf);
-	explain_outNode(&buf,
-					queryDesc->plannedstmt->planTree, queryDesc->planstate,
-					NULL, 0, es);
+	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
 
 	/*
 	 * If we ran the command, run any AFTER triggers it queued.  (Note this
@@ -290,7 +280,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	}
 
 	/* Print info about runtime of triggers */
-	if (es->printAnalyze)
+	if (stmt->analyze)
 	{
 		ResultRelInfo *rInfo;
 		bool		show_relname;
@@ -335,7 +325,26 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	do_text_output_multiline(tstate, buf.data);
 
 	pfree(buf.data);
-	pfree(es);
+}
+
+/*
+ * ExplainOneResult -
+ *	  converts a Plan node into ascii string and appends it to 'str'
+ */
+void
+ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+				 bool analyze, bool verbose)
+{
+	ExplainState	es = { 0 };
+
+	es.printTList = verbose;
+	es.printAnalyze = analyze;
+	es.pstmt = queryDesc->plannedstmt;
+	es.rtable = queryDesc->plannedstmt->rtable;
+
+	explain_outNode(str,
+					queryDesc->plannedstmt->planTree, queryDesc->planstate,
+					NULL, 0, &es);
 }
 
 /*
diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c
index a933d3c..ff2df9b 100644
--- a/src/backend/tcop/pquery.c
+++ b/src/backend/tcop/pquery.c
@@ -32,7 +32,7 @@
  * if there are several).
  */
 Portal		ActivePortal = NULL;
-
+bool		force_instrument = false;
 
 static void ProcessQuery(PlannedStmt *plan,
 			 ParamListInfo params,
@@ -76,7 +76,7 @@ CreateQueryDesc(PlannedStmt *plannedstmt,
 	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
 	qd->dest = dest;			/* output dest */
 	qd->params = params;		/* parameter values passed into query */
-	qd->doInstrument = doInstrument;	/* instrumentation wanted? */
+	qd->doInstrument = force_instrument || doInstrument;	/* instrumentation wanted? */
 
 	/* null these fields until set by ExecutorStart */
 	qd->tupDesc = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 58851e5..015f06d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -169,6 +169,7 @@ static const char *assign_pgstat_temp_directory(const char *newval, bool doit, G
 
 static char *config_enum_get_options(struct config_enum *record, 
 									 const char *prefix, const char *suffix);
+static void initialize_option(struct config_generic *gconf);
 
 
 /*
@@ -3177,112 +3178,7 @@ InitializeGUCOptions(void)
 	for (i = 0; i < num_guc_variables; i++)
 	{
 		struct config_generic *gconf = guc_variables[i];
-
-		gconf->status = 0;
-		gconf->reset_source = PGC_S_DEFAULT;
-		gconf->source = PGC_S_DEFAULT;
-		gconf->stack = NULL;
-		gconf->sourcefile = NULL;
-		gconf->sourceline = 0;
-
-		switch (gconf->vartype)
-		{
-			case PGC_BOOL:
-				{
-					struct config_bool *conf = (struct config_bool *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, (int) conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_INT:
-				{
-					struct config_int *conf = (struct config_int *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_REAL:
-				{
-					struct config_real *conf = (struct config_real *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %g",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_STRING:
-				{
-					struct config_string *conf = (struct config_string *) gconf;
-					char	   *str;
-
-					*conf->variable = NULL;
-					conf->reset_val = NULL;
-
-					if (conf->boot_val == NULL)
-					{
-						/* leave the value NULL, do not call assign hook */
-						break;
-					}
-
-					str = guc_strdup(FATAL, conf->boot_val);
-					conf->reset_val = str;
-
-					if (conf->assign_hook)
-					{
-						const char *newstr;
-
-						newstr = (*conf->assign_hook) (str, true,
-													   PGC_S_DEFAULT);
-						if (newstr == NULL)
-						{
-							elog(FATAL, "failed to initialize %s to \"%s\"",
-								 conf->gen.name, str);
-						}
-						else if (newstr != str)
-						{
-							free(str);
-
-							/*
-							 * See notes in set_config_option about casting
-							 */
-							str = (char *) newstr;
-							conf->reset_val = str;
-						}
-					}
-					*conf->variable = str;
-					break;
-				}
-			case PGC_ENUM:
-				{
-					struct config_enum *conf = (struct config_enum *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %s",
-								 conf->gen.name, 
-								 config_enum_lookup_by_value(conf, conf->boot_val));
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-		}
+		initialize_option(gconf);
 	}
 
 	guc_dirty = false;
@@ -3338,6 +3234,115 @@ InitializeGUCOptions(void)
 	}
 }
 
+static void
+initialize_option(struct config_generic *gconf)
+{
+	gconf->status = 0;
+	gconf->reset_source = PGC_S_DEFAULT;
+	gconf->source = PGC_S_DEFAULT;
+	gconf->stack = NULL;
+	gconf->sourcefile = NULL;
+	gconf->sourceline = 0;
+
+	switch (gconf->vartype)
+	{
+		case PGC_BOOL:
+			{
+				struct config_bool *conf = (struct config_bool *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, (int) conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_INT:
+			{
+				struct config_int *conf = (struct config_int *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_REAL:
+			{
+				struct config_real *conf = (struct config_real *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %g",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_STRING:
+			{
+				struct config_string *conf = (struct config_string *) gconf;
+				char	   *str;
+
+				*conf->variable = NULL;
+				conf->reset_val = NULL;
+
+				if (conf->boot_val == NULL)
+				{
+					/* leave the value NULL, do not call assign hook */
+					break;
+				}
+
+				str = guc_strdup(FATAL, conf->boot_val);
+				conf->reset_val = str;
+
+				if (conf->assign_hook)
+				{
+					const char *newstr;
+
+					newstr = (*conf->assign_hook) (str, true,
+												   PGC_S_DEFAULT);
+					if (newstr == NULL)
+					{
+						elog(FATAL, "failed to initialize %s to \"%s\"",
+							 conf->gen.name, str);
+					}
+					else if (newstr != str)
+					{
+						free(str);
+
+						/*
+						 * See notes in set_config_option about casting
+						 */
+						str = (char *) newstr;
+						conf->reset_val = str;
+					}
+				}
+				*conf->variable = str;
+				break;
+			}
+		case PGC_ENUM:
+			{
+				struct config_enum *conf = (struct config_enum *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %s",
+							 conf->gen.name, 
+							 config_enum_lookup_by_value(conf, conf->boot_val));
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+	}
+}
 
 /*
  * Select the configuration files and data directory to be used, and
@@ -5639,6 +5644,7 @@ define_custom_variable(struct config_generic * variable)
 	if (res == NULL)
 	{
 		/* No placeholder to replace, so just add it */
+		initialize_option(variable);
 		add_guc_variable(variable, ERROR);
 		return;
 	}
@@ -5673,6 +5679,8 @@ define_custom_variable(struct config_generic * variable)
 		set_config_option(name, value,
 						  pHolder->gen.context, pHolder->gen.source,
 						  GUC_ACTION_SET, true);
+	else
+		initialize_option(variable);
 
 	/*
 	 * Free up as much as we conveniently can of the placeholder structure
@@ -5806,6 +5814,29 @@ DefineCustomEnumVariable(const char *name,
 	define_custom_variable(&var->gen);
 }
 
+static const int config_varsize[] =
+{
+	sizeof(struct config_bool),
+	sizeof(struct config_int),
+	sizeof(struct config_real),
+	sizeof(struct config_string),
+	sizeof(struct config_enum),
+};
+
+void
+DefineCustomVariable(enum config_type type, const void *variable)
+{
+	int		size = config_varsize[type];
+	const struct config_generic	   *var = variable;
+	struct config_generic		   *gen;
+
+	gen = (struct config_generic *) guc_malloc(ERROR, size);
+	memcpy(gen, var, size);
+	gen->name = guc_strdup(ERROR, var->name);
+	gen->vartype = type;
+	define_custom_variable(gen);
+}
+
 void
 EmitWarningsOnPlaceholders(const char *className)
 {
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 5b808d5..cf07915 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -41,4 +41,7 @@ extern void ExplainOneUtility(Node *utilityStmt, ExplainStmt *stmt,
 extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 			   ExplainStmt *stmt, TupOutputState *tstate);
 
+extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+							 bool analyze, bool verbose);
+
 #endif   /* EXPLAIN_H */
diff --git a/src/include/executor/execdesc.h b/src/include/executor/execdesc.h
index 2e2c159..854721c 100644
--- a/src/include/executor/execdesc.h
+++ b/src/include/executor/execdesc.h
@@ -20,6 +20,8 @@
 #include "tcop/dest.h"
 
 
+extern PGDLLIMPORT bool force_instrument;
+
 /* ----------------
  *		query descriptor:
  *
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index 8791660..3982a27 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -242,5 +242,6 @@ extern const char *config_enum_lookup_by_value(struct config_enum *record, int v
 extern bool config_enum_lookup_by_name(struct config_enum *record,
 									  const char *value, int *retval);
 
+extern void DefineCustomVariable(enum config_type type, const void *variable);
 
 #endif   /* GUC_TABLES_H */
#49Jeff Davis
pgsql@j-davis.com
In reply to: Jeff Davis (#48)
Re: auto_explain contrib moudle

On Sat, 2008-11-08 at 11:32 -0800, Jeff Davis wrote:

One thing I'm unsure of (this question is for ITAGAKI Takahiro): why is
it necessary to define a new function DefineCustomVariable(), when there
are already functions DefineCustomBoolVariable() and
DefineCustomIntVariable()?

Oh, I see. It's needed to set the flags to GUC_UNIT_MS.

Another question: what is explain.log_analyze supposed to do? Changing
it doesn't seem to have an effect; it always prints out the actual time.

Regards,
Jeff Davis

#50ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Jeff Davis (#49)
1 attachment(s)
Re: auto_explain contrib moudle

Thank you for reviewing.

Jeff Davis <pgsql@j-davis.com> wrote:

Another question: what is explain.log_analyze supposed to do? Changing
it doesn't seem to have an effect; it always prints out the actual time.

That's because explain.log_analyze requires executor instruments,
and it's not free. I think we'd better to have an option to avoid
the overheads... Oops, I found my bug when force_instrument is
turned on. It should be enabled only when
(explain_log_min_duration >= 0 && explain_log_analyze).

I send a new patch to fix it. A documentation about
explain.log_nested_statements is also added to the sgml file.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Attachments:

auto-explain.1110.patchapplication/octet-stream; name=auto-explain.1110.patchDownload
diff -cprN HEAD/contrib/Makefile auto-explain/contrib/Makefile
*** HEAD/contrib/Makefile	Wed Jul 30 03:31:20 2008
--- auto-explain/contrib/Makefile	Mon Nov 10 17:55:49 2008
*************** include $(top_builddir)/src/Makefile.glo
*** 6,11 ****
--- 6,12 ----
  
  WANTED_DIRS = \
  		adminpack	\
+ 		auto_explain	\
  		btree_gist	\
  		chkpass		\
  		citext		\
diff -cprN HEAD/contrib/auto_explain/Makefile auto-explain/contrib/auto_explain/Makefile
*** HEAD/contrib/auto_explain/Makefile	Thu Jan  1 09:00:00 1970
--- auto-explain/contrib/auto_explain/Makefile	Mon Nov 10 17:29:01 2008
***************
*** 0 ****
--- 1,13 ----
+ MODULE_big = auto_explain
+ OBJS = auto_explain.o
+ 
+ ifdef USE_PGXS
+ PG_CONFIG = pg_config
+ PGXS := $(shell $(PG_CONFIG) --pgxs)
+ include $(PGXS)
+ else
+ subdir = contrib/auto_explain
+ top_builddir = ../..
+ include $(top_builddir)/src/Makefile.global
+ include $(top_srcdir)/contrib/contrib-global.mk
+ endif
diff -cprN HEAD/contrib/auto_explain/auto_explain.c auto-explain/contrib/auto_explain/auto_explain.c
*** HEAD/contrib/auto_explain/auto_explain.c	Thu Jan  1 09:00:00 1970
--- auto-explain/contrib/auto_explain/auto_explain.c	Mon Nov 10 17:52:03 2008
***************
*** 0 ****
--- 1,215 ----
+ /*
+  * auto_explain.c
+  */
+ #include "postgres.h"
+ 
+ #include "commands/explain.h"
+ #include "executor/instrument.h"
+ #include "miscadmin.h"
+ #include "utils/guc_tables.h"
+ 
+ PG_MODULE_MAGIC;
+ 
+ #define GUCNAME(name)		("explain." name)
+ 
+ static int		explain_log_min_duration = -1;	/* msec or -1 */
+ static bool		explain_log_analyze = false;
+ static bool		explain_log_verbose = false;
+ static bool		explain_log_nested = false;
+ 
+ static bool		toplevel = true;
+ static ExecutorRun_hook_type	prev_ExecutorRun = NULL;
+ 
+ void	_PG_init(void);
+ void	_PG_fini(void);
+ 
+ static void explain_ExecutorRun(QueryDesc *queryDesc,
+ 								ScanDirection direction,
+ 								long count);
+ static bool assign_log_min_duration(int newval, bool doit, GucSource source);
+ static bool assign_log_analyze(bool newval, bool doit, GucSource source);
+ static bool assign_log_verbose(bool newval, bool doit, GucSource source);
+ static bool assign_log_nested(bool newval, bool doit, GucSource source);
+ 
+ static struct config_int def_log_min_duration =
+ {
+ 	{
+ 		GUCNAME("log_min_duration"),
+ 		PGC_USERSET,
+ 		STATS_MONITORING,
+ 		"Sets the minimum execution time above which plans will be logged.",
+ 		"Zero prints all plans. -1 turns this feature off.",
+ 		GUC_UNIT_MS
+ 	},
+ 	&explain_log_min_duration,
+ 	-1, -1, INT_MAX / 1000, assign_log_min_duration, NULL
+ };
+ 
+ static struct config_bool def_log_analyze =
+ {
+ 	{
+ 		GUCNAME("log_analyze"),
+ 		PGC_USERSET,
+ 		STATS_MONITORING,
+ 		"Use EXPLAIN ANALYZE for plan logging."
+ 	},
+ 	&explain_log_analyze,
+ 	false, assign_log_analyze, NULL
+ };
+ 
+ static struct config_bool def_log_verbose =
+ {
+ 	{
+ 		GUCNAME("log_verbose"),
+ 		PGC_USERSET,
+ 		STATS_MONITORING,
+ 		"Use EXPLAIN VERBOSE for plan logging."
+ 	},
+ 	&explain_log_verbose,
+ 	false, assign_log_verbose, NULL
+ };
+ 
+ static struct config_bool def_log_nested_statements =
+ {
+ 	{
+ 		GUCNAME("log_nested_statements"),
+ 		PGC_USERSET,
+ 		STATS_MONITORING,
+ 		"Log nested statements."
+ 	},
+ 	&explain_log_nested,
+ 	false, assign_log_nested, NULL
+ };
+ 
+ void
+ _PG_init(void)
+ {
+ 	DefineCustomVariable(PGC_INT, &def_log_min_duration);
+ 	DefineCustomVariable(PGC_BOOL, &def_log_analyze);
+ 	DefineCustomVariable(PGC_BOOL, &def_log_verbose);
+ 	DefineCustomVariable(PGC_BOOL, &def_log_nested_statements);
+ 
+ 	/* install ExecutorRun_hook */
+ 	prev_ExecutorRun = ExecutorRun_hook;
+ 	ExecutorRun_hook = explain_ExecutorRun;
+ }
+ 
+ void
+ _PG_fini(void)
+ {
+ 	/* uninstall ExecutorRun_hook */
+ 	ExecutorRun_hook = prev_ExecutorRun;
+ }
+ 
+ void
+ explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
+ {
+ 	if ((toplevel || explain_log_nested) && explain_log_min_duration >= 0)
+ 	{
+ 		instr_time		starttime;
+ 		instr_time		duration;
+ 		double			msec;
+ 
+ 		/* Disable our hooks temporarily during the top-level query. */
+ 		toplevel = false;
+ 		PG_TRY();
+ 		{
+ 			INSTR_TIME_SET_CURRENT(starttime);
+ 
+ 			if (prev_ExecutorRun)
+ 				prev_ExecutorRun(queryDesc, direction, count);
+ 			else
+ 				standard_ExecutorRun(queryDesc, direction, count);
+ 
+ 			INSTR_TIME_SET_CURRENT(duration);
+ 			INSTR_TIME_SUBTRACT(duration, starttime);
+ 			msec = INSTR_TIME_GET_MILLISEC(duration);
+ 
+ 			/* Log plan if duration is exceeded. */
+ 			if (msec > explain_log_min_duration)
+ 			{
+ 				StringInfoData	buf;
+ 
+ 				initStringInfo(&buf);
+ 				ExplainOneResult(&buf, queryDesc,
+ 					queryDesc->doInstrument && explain_log_analyze,
+ 					explain_log_verbose);
+ 
+ 				/* Remove last line break */
+ 				if (buf.len > 0 && buf.data[buf.len - 1] == '\n')
+ 				{
+ 					buf.data[buf.len - 1] = '\0';
+ 					buf.len--;
+ 				}
+ 				ereport(LOG,
+ 						(errmsg("duration: %.3f ms  plan:\n%s",
+ 								msec, buf.data)));
+ 
+ 				pfree(buf.data);
+ 			}
+ 
+ 			toplevel = true;
+ 		}
+ 		PG_CATCH();
+ 		{
+ 			toplevel = true;
+ 			PG_RE_THROW();
+ 		}
+ 		PG_END_TRY();
+ 	}
+ 	else
+ 	{
+ 		/* ignore recursive executions, that are typically function calls */
+ 		if (prev_ExecutorRun)
+ 			prev_ExecutorRun(queryDesc, direction, count);
+ 		else
+ 			standard_ExecutorRun(queryDesc, direction, count);
+ 	}
+ }
+ 
+ /* Emulate PGC_SUSET for custom variables. */
+ static bool
+ suset_assign(GucSource source, const char *name)
+ {
+ 	if (source >= PGC_S_CLIENT && !superuser())
+ 	{
+ 		ereport(GUC_complaint_elevel(source),
+ 				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ 				 errmsg("permission denied to set parameter \"%s\"", name)));
+ 		return false;
+ 	}
+ 
+ 	return true;
+ }
+ 
+ static bool
+ assign_log_min_duration(int newval, bool doit, GucSource source)
+ {
+ 	if (!suset_assign(source, GUCNAME("log_min_duration")))
+ 		return false;
+ 	if (doit)
+ 		force_instrument = (newval >= 0 && explain_log_analyze);
+ 	return true;
+ }
+ 
+ static bool
+ assign_log_analyze(bool newval, bool doit, GucSource source)
+ {
+ 	if (!suset_assign(source, GUCNAME("log_analyze")))
+ 		return false;
+ 	if (doit)
+ 		force_instrument = (explain_log_min_duration >= 0 && newval);
+ 	return true;
+ }
+ 
+ static bool
+ assign_log_verbose(bool newval, bool doit, GucSource source)
+ {
+ 	return suset_assign(source, GUCNAME("log_verbose"));
+ }
+ 
+ static bool
+ assign_log_nested(bool newval, bool doit, GucSource source)
+ {
+ 	return suset_assign(source, GUCNAME("log_nested_statements"));
+ }
diff -cprN HEAD/doc/src/sgml/auto-explain.sgml auto-explain/doc/src/sgml/auto-explain.sgml
*** HEAD/doc/src/sgml/auto-explain.sgml	Thu Jan  1 09:00:00 1970
--- auto-explain/doc/src/sgml/auto-explain.sgml	Mon Nov 10 17:43:33 2008
***************
*** 0 ****
--- 1,150 ----
+ <sect1 id="autoexplain">
+  <title>auto_explain</title>
+ 
+  <indexterm zone="autoexplain">
+   <primary>auto_explain</primary>
+  </indexterm>
+ 
+  <para>
+   The <filename>auto_explain</filename> module provides a means for
+   logging execution plans that execution times are longer than configuration.
+  </para>
+ 
+  <para>
+   You can <command>LOAD</> this module dynamically or preload it automatically with
+   <varname>shared_preload_libraries</> or <varname>local_preload_libraries</>.
+  </para>
+ 
+  <sect2>
+   <title>The <structname>auto_explain</structname> view</title>
+ 
+   <para>
+    The definitions of the columns exposed by the view are:
+   </para>
+ 
+   <para>
+    There is one row for each statement. Statements are grouped when they have
+    same SQL text, are in the same database, and are executed by the same user.
+   </para>
+ 
+   <para>
+    Because of security restriction, non-super users cannot see query strings
+    executed by other users.
+   </para>
+  </sect2>
+ 
+  <sect2>
+   <title>Configuration parameters</title>
+ 
+   <variablelist>
+    <varlistentry>
+     <term>
+      <varname>explain.log_min_duration</varname> (<type>integer</type>)
+     </term>
+     <listitem>
+      <para>
+       <varname>explain.log_min_duration</varname> is the minimum execution time
+       in milliseconds which execution plan will be logged. Setting this to zero
+       prints all plans. Minus-one (the default) disables logging plans.
+       For example, if you set it to <literal>250ms</literal> then all plan
+       that run 250ms or longer in executor will be logged.
+       Enabling this parameter can be helpful in tracking down unoptimized queries
+       in your applications. Only superusers can change this setting.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
+     <term>
+      <varname>explain.log_analyze</varname> (<type>boolean</type>)
+     </term>
+ 
+     <listitem>
+      <para>
+       <varname>explain.log_analyze</varname> enables to use EXPLAIN ANALYZE when
+       an execution plan is logged by <varname>explain.log_min_duration</varname>.
+       This parameter is off by default. Only superusers can change this setting.
+      </para>
+      <para>
+       NOTE: If you set the parameter on, instrument timers are enabled even if
+       you don't use EXPLAIN ANALYZE. This has some overhead to execute plans.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
+     <term>
+      <varname>explain.log_verbose</varname> (<type>boolean</type>)
+     </term>
+ 
+     <listitem>
+      <para>
+       <varname>explain.log_verbose</varname> enables to use EXPLAIN VERBOSE when
+       an execution plan is logged by <varname>explain.log_min_duration</varname>.
+       This parameter is off by default. Only superusers can change this setting.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
+     <term>
+      <varname>explain.log_nested_statements</varname> (<type>boolean</type>)
+     </term>
+ 
+     <listitem>
+      <para>
+       <varname>explain.log_nested_statements</varname> enables to log plans of
+       nested statements. If the value if off, only top-level plans are logged.
+       This parameter is off by default. Only superusers can change this setting.
+      </para>
+     </listitem>
+    </varlistentry>
+   </variablelist>
+ 
+   <para>
+    If you set these explain.* parameters in your postgresql.conf,
+    you also need to add 'explain' in <varname>custom_variable_classes</>.
+   </para>
+ 
+   <programlisting>
+   # postgresql.conf
+   shared_preload_libraries = 'auto_explain'
+   
+   custom_variable_classes = 'explain'
+   explain.log_min_duration = 3s
+   </programlisting>
+  </sect2>
+ 
+  <sect2>
+   <title>Sample output</title>
+ 
+   <programlisting>
+   postgres=# LOAD 'auto_explain';
+   postgres=# SET explain.log_min_duration = 0;
+   postgres=# SELECT count(*)
+                FROM pg_class, pg_index
+               WHERE oid = indrelid AND indisunique;
+   
+   LOG:  duration: 0.986 ms  plan:
+           Aggregate  (cost=14.90..14.91 rows=1 width=0)
+             ->  Hash Join  (cost=3.91..14.70 rows=81 width=0)
+                   Hash Cond: (pg_class.oid = pg_index.indrelid)
+                   ->  Seq Scan on pg_class  (cost=0.00..8.27 rows=227 width=4)
+                   ->  Hash  (cost=2.90..2.90 rows=81 width=4)
+                         ->  Seq Scan on pg_index  (cost=0.00..2.90 rows=81 width=4)
+                               Filter: indisunique
+   STATEMENT:  SELECT count(*)
+             FROM pg_class, pg_index
+            WHERE oid = indrelid AND indisunique;
+   </programlisting>
+  </sect2>
+ 
+  <sect2>
+   <title>Authors</title>
+ 
+   <para>
+    Takahiro Itagaki <email>itagaki.takahiro@oss.ntt.co.jp</email>
+   </para>
+  </sect2>
+ 
+ </sect1>
diff -cprN HEAD/doc/src/sgml/contrib.sgml auto-explain/doc/src/sgml/contrib.sgml
*** HEAD/doc/src/sgml/contrib.sgml	Wed Jul 30 03:31:20 2008
--- auto-explain/doc/src/sgml/contrib.sgml	Mon Nov 10 17:55:49 2008
*************** psql -d dbname -f <replaceable>SHAREDIR<
*** 79,84 ****
--- 79,85 ----
   </para>
  
   &adminpack;
+  &auto-explain;
   &btree-gist;
   &chkpass;
   &citext;
diff -cprN HEAD/doc/src/sgml/filelist.sgml auto-explain/doc/src/sgml/filelist.sgml
*** HEAD/doc/src/sgml/filelist.sgml	Wed Jul 30 03:31:20 2008
--- auto-explain/doc/src/sgml/filelist.sgml	Mon Nov 10 17:55:49 2008
***************
*** 92,97 ****
--- 92,98 ----
  <!-- contrib information -->
  <!entity contrib         SYSTEM "contrib.sgml">
  <!entity adminpack       SYSTEM "adminpack.sgml">
+ <!entity auto-explain    SYSTEM "auto-explain.sgml">
  <!entity btree-gist      SYSTEM "btree-gist.sgml">
  <!entity chkpass         SYSTEM "chkpass.sgml">
  <!entity citext          SYSTEM "citext.sgml">
diff -cprN HEAD/src/backend/commands/explain.c auto-explain/src/backend/commands/explain.c
*** HEAD/src/backend/commands/explain.c	Tue Oct  7 05:29:38 2008
--- auto-explain/src/backend/commands/explain.c	Mon Nov 10 17:55:49 2008
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 224,230 ****
  	QueryDesc  *queryDesc;
  	instr_time	starttime;
  	double		totaltime = 0;
- 	ExplainState *es;
  	StringInfoData buf;
  	int			eflags;
  
--- 224,229 ----
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 265,281 ****
  		totaltime += elapsed_time(&starttime);
  	}
  
- 	es = (ExplainState *) palloc0(sizeof(ExplainState));
- 
- 	es->printTList = stmt->verbose;
- 	es->printAnalyze = stmt->analyze;
- 	es->pstmt = queryDesc->plannedstmt;
- 	es->rtable = queryDesc->plannedstmt->rtable;
- 
  	initStringInfo(&buf);
! 	explain_outNode(&buf,
! 					queryDesc->plannedstmt->planTree, queryDesc->planstate,
! 					NULL, 0, es);
  
  	/*
  	 * If we ran the command, run any AFTER triggers it queued.  (Note this
--- 264,271 ----
  		totaltime += elapsed_time(&starttime);
  	}
  
  	initStringInfo(&buf);
! 	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
  
  	/*
  	 * If we ran the command, run any AFTER triggers it queued.  (Note this
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 290,296 ****
  	}
  
  	/* Print info about runtime of triggers */
! 	if (es->printAnalyze)
  	{
  		ResultRelInfo *rInfo;
  		bool		show_relname;
--- 280,286 ----
  	}
  
  	/* Print info about runtime of triggers */
! 	if (stmt->analyze)
  	{
  		ResultRelInfo *rInfo;
  		bool		show_relname;
*************** ExplainOnePlan(PlannedStmt *plannedstmt,
*** 335,341 ****
  	do_text_output_multiline(tstate, buf.data);
  
  	pfree(buf.data);
! 	pfree(es);
  }
  
  /*
--- 325,350 ----
  	do_text_output_multiline(tstate, buf.data);
  
  	pfree(buf.data);
! }
! 
! /*
!  * ExplainOneResult -
!  *	  converts a Plan node into ascii string and appends it to 'str'
!  */
! void
! ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
! 				 bool analyze, bool verbose)
! {
! 	ExplainState	es = { 0 };
! 
! 	es.printTList = verbose;
! 	es.printAnalyze = analyze;
! 	es.pstmt = queryDesc->plannedstmt;
! 	es.rtable = queryDesc->plannedstmt->rtable;
! 
! 	explain_outNode(str,
! 					queryDesc->plannedstmt->planTree, queryDesc->planstate,
! 					NULL, 0, &es);
  }
  
  /*
diff -cprN HEAD/src/backend/tcop/pquery.c auto-explain/src/backend/tcop/pquery.c
*** HEAD/src/backend/tcop/pquery.c	Fri Aug  1 22:16:09 2008
--- auto-explain/src/backend/tcop/pquery.c	Mon Nov 10 17:55:49 2008
***************
*** 32,38 ****
   * if there are several).
   */
  Portal		ActivePortal = NULL;
! 
  
  static void ProcessQuery(PlannedStmt *plan,
  			 ParamListInfo params,
--- 32,38 ----
   * if there are several).
   */
  Portal		ActivePortal = NULL;
! bool		force_instrument = false;
  
  static void ProcessQuery(PlannedStmt *plan,
  			 ParamListInfo params,
*************** CreateQueryDesc(PlannedStmt *plannedstmt
*** 76,82 ****
  	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
  	qd->dest = dest;			/* output dest */
  	qd->params = params;		/* parameter values passed into query */
! 	qd->doInstrument = doInstrument;	/* instrumentation wanted? */
  
  	/* null these fields until set by ExecutorStart */
  	qd->tupDesc = NULL;
--- 76,82 ----
  	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
  	qd->dest = dest;			/* output dest */
  	qd->params = params;		/* parameter values passed into query */
! 	qd->doInstrument = force_instrument || doInstrument;	/* instrumentation wanted? */
  
  	/* null these fields until set by ExecutorStart */
  	qd->tupDesc = NULL;
diff -cprN HEAD/src/backend/utils/misc/guc.c auto-explain/src/backend/utils/misc/guc.c
*** HEAD/src/backend/utils/misc/guc.c	Sun Nov  9 09:28:35 2008
--- auto-explain/src/backend/utils/misc/guc.c	Mon Nov 10 17:55:49 2008
*************** static const char *assign_pgstat_temp_di
*** 169,174 ****
--- 169,175 ----
  
  static char *config_enum_get_options(struct config_enum *record, 
  									 const char *prefix, const char *suffix);
+ static void initialize_option(struct config_generic *gconf);
  
  
  /*
*************** InitializeGUCOptions(void)
*** 3194,3305 ****
  	for (i = 0; i < num_guc_variables; i++)
  	{
  		struct config_generic *gconf = guc_variables[i];
! 
! 		gconf->status = 0;
! 		gconf->reset_source = PGC_S_DEFAULT;
! 		gconf->source = PGC_S_DEFAULT;
! 		gconf->stack = NULL;
! 		gconf->sourcefile = NULL;
! 		gconf->sourceline = 0;
! 
! 		switch (gconf->vartype)
! 		{
! 			case PGC_BOOL:
! 				{
! 					struct config_bool *conf = (struct config_bool *) gconf;
! 
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %d",
! 								 conf->gen.name, (int) conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_INT:
! 				{
! 					struct config_int *conf = (struct config_int *) gconf;
! 
! 					Assert(conf->boot_val >= conf->min);
! 					Assert(conf->boot_val <= conf->max);
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %d",
! 								 conf->gen.name, conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_REAL:
! 				{
! 					struct config_real *conf = (struct config_real *) gconf;
! 
! 					Assert(conf->boot_val >= conf->min);
! 					Assert(conf->boot_val <= conf->max);
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %g",
! 								 conf->gen.name, conf->boot_val);
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 			case PGC_STRING:
! 				{
! 					struct config_string *conf = (struct config_string *) gconf;
! 					char	   *str;
! 
! 					*conf->variable = NULL;
! 					conf->reset_val = NULL;
! 
! 					if (conf->boot_val == NULL)
! 					{
! 						/* leave the value NULL, do not call assign hook */
! 						break;
! 					}
! 
! 					str = guc_strdup(FATAL, conf->boot_val);
! 					conf->reset_val = str;
! 
! 					if (conf->assign_hook)
! 					{
! 						const char *newstr;
! 
! 						newstr = (*conf->assign_hook) (str, true,
! 													   PGC_S_DEFAULT);
! 						if (newstr == NULL)
! 						{
! 							elog(FATAL, "failed to initialize %s to \"%s\"",
! 								 conf->gen.name, str);
! 						}
! 						else if (newstr != str)
! 						{
! 							free(str);
! 
! 							/*
! 							 * See notes in set_config_option about casting
! 							 */
! 							str = (char *) newstr;
! 							conf->reset_val = str;
! 						}
! 					}
! 					*conf->variable = str;
! 					break;
! 				}
! 			case PGC_ENUM:
! 				{
! 					struct config_enum *conf = (struct config_enum *) gconf;
! 
! 					if (conf->assign_hook)
! 						if (!(*conf->assign_hook) (conf->boot_val, true,
! 												   PGC_S_DEFAULT))
! 							elog(FATAL, "failed to initialize %s to %s",
! 								 conf->gen.name, 
! 								 config_enum_lookup_by_value(conf, conf->boot_val));
! 					*conf->variable = conf->reset_val = conf->boot_val;
! 					break;
! 				}
! 		}
  	}
  
  	guc_dirty = false;
--- 3195,3201 ----
  	for (i = 0; i < num_guc_variables; i++)
  	{
  		struct config_generic *gconf = guc_variables[i];
! 		initialize_option(gconf);
  	}
  
  	guc_dirty = false;
*************** InitializeGUCOptions(void)
*** 3355,3360 ****
--- 3251,3365 ----
  	}
  }
  
+ static void
+ initialize_option(struct config_generic *gconf)
+ {
+ 	gconf->status = 0;
+ 	gconf->reset_source = PGC_S_DEFAULT;
+ 	gconf->source = PGC_S_DEFAULT;
+ 	gconf->stack = NULL;
+ 	gconf->sourcefile = NULL;
+ 	gconf->sourceline = 0;
+ 
+ 	switch (gconf->vartype)
+ 	{
+ 		case PGC_BOOL:
+ 			{
+ 				struct config_bool *conf = (struct config_bool *) gconf;
+ 
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %d",
+ 							 conf->gen.name, (int) conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_INT:
+ 			{
+ 				struct config_int *conf = (struct config_int *) gconf;
+ 
+ 				Assert(conf->boot_val >= conf->min);
+ 				Assert(conf->boot_val <= conf->max);
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %d",
+ 							 conf->gen.name, conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_REAL:
+ 			{
+ 				struct config_real *conf = (struct config_real *) gconf;
+ 
+ 				Assert(conf->boot_val >= conf->min);
+ 				Assert(conf->boot_val <= conf->max);
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %g",
+ 							 conf->gen.name, conf->boot_val);
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 		case PGC_STRING:
+ 			{
+ 				struct config_string *conf = (struct config_string *) gconf;
+ 				char	   *str;
+ 
+ 				*conf->variable = NULL;
+ 				conf->reset_val = NULL;
+ 
+ 				if (conf->boot_val == NULL)
+ 				{
+ 					/* leave the value NULL, do not call assign hook */
+ 					break;
+ 				}
+ 
+ 				str = guc_strdup(FATAL, conf->boot_val);
+ 				conf->reset_val = str;
+ 
+ 				if (conf->assign_hook)
+ 				{
+ 					const char *newstr;
+ 
+ 					newstr = (*conf->assign_hook) (str, true,
+ 												   PGC_S_DEFAULT);
+ 					if (newstr == NULL)
+ 					{
+ 						elog(FATAL, "failed to initialize %s to \"%s\"",
+ 							 conf->gen.name, str);
+ 					}
+ 					else if (newstr != str)
+ 					{
+ 						free(str);
+ 
+ 						/*
+ 						 * See notes in set_config_option about casting
+ 						 */
+ 						str = (char *) newstr;
+ 						conf->reset_val = str;
+ 					}
+ 				}
+ 				*conf->variable = str;
+ 				break;
+ 			}
+ 		case PGC_ENUM:
+ 			{
+ 				struct config_enum *conf = (struct config_enum *) gconf;
+ 
+ 				if (conf->assign_hook)
+ 					if (!(*conf->assign_hook) (conf->boot_val, true,
+ 											   PGC_S_DEFAULT))
+ 						elog(FATAL, "failed to initialize %s to %s",
+ 							 conf->gen.name, 
+ 							 config_enum_lookup_by_value(conf, conf->boot_val));
+ 				*conf->variable = conf->reset_val = conf->boot_val;
+ 				break;
+ 			}
+ 	}
+ }
  
  /*
   * Select the configuration files and data directory to be used, and
*************** define_custom_variable(struct config_gen
*** 5656,5661 ****
--- 5661,5667 ----
  	if (res == NULL)
  	{
  		/* No placeholder to replace, so just add it */
+ 		initialize_option(variable);
  		add_guc_variable(variable, ERROR);
  		return;
  	}
*************** define_custom_variable(struct config_gen
*** 5690,5695 ****
--- 5696,5703 ----
  		set_config_option(name, value,
  						  pHolder->gen.context, pHolder->gen.source,
  						  GUC_ACTION_SET, true);
+ 	else
+ 		initialize_option(variable);
  
  	/*
  	 * Free up as much as we conveniently can of the placeholder structure
*************** DefineCustomEnumVariable(const char *nam
*** 5821,5826 ****
--- 5829,5857 ----
  	var->assign_hook = assign_hook;
  	var->show_hook = show_hook;
  	define_custom_variable(&var->gen);
+ }
+ 
+ static const int config_varsize[] =
+ {
+ 	sizeof(struct config_bool),
+ 	sizeof(struct config_int),
+ 	sizeof(struct config_real),
+ 	sizeof(struct config_string),
+ 	sizeof(struct config_enum),
+ };
+ 
+ void
+ DefineCustomVariable(enum config_type type, const void *variable)
+ {
+ 	int		size = config_varsize[type];
+ 	const struct config_generic	   *var = variable;
+ 	struct config_generic		   *gen;
+ 
+ 	gen = (struct config_generic *) guc_malloc(ERROR, size);
+ 	memcpy(gen, var, size);
+ 	gen->name = guc_strdup(ERROR, var->name);
+ 	gen->vartype = type;
+ 	define_custom_variable(gen);
  }
  
  void
diff -cprN HEAD/src/include/commands/explain.h auto-explain/src/include/commands/explain.h
*** HEAD/src/include/commands/explain.h	Wed Jan  2 04:45:57 2008
--- auto-explain/src/include/commands/explain.h	Mon Nov 10 17:55:49 2008
*************** extern void ExplainOneUtility(Node *util
*** 41,44 ****
--- 41,47 ----
  extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
  			   ExplainStmt *stmt, TupOutputState *tstate);
  
+ extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+ 							 bool analyze, bool verbose);
+ 
  #endif   /* EXPLAIN_H */
diff -cprN HEAD/src/include/executor/execdesc.h auto-explain/src/include/executor/execdesc.h
*** HEAD/src/include/executor/execdesc.h	Wed Jan  2 04:45:57 2008
--- auto-explain/src/include/executor/execdesc.h	Mon Nov 10 17:55:49 2008
***************
*** 20,25 ****
--- 20,27 ----
  #include "tcop/dest.h"
  
  
+ extern PGDLLIMPORT bool force_instrument;
+ 
  /* ----------------
   *		query descriptor:
   *
diff -cprN HEAD/src/include/utils/guc_tables.h auto-explain/src/include/utils/guc_tables.h
*** HEAD/src/include/utils/guc_tables.h	Tue Sep 30 19:52:14 2008
--- auto-explain/src/include/utils/guc_tables.h	Mon Nov 10 17:55:49 2008
*************** extern const char *config_enum_lookup_by
*** 242,246 ****
--- 242,247 ----
  extern bool config_enum_lookup_by_name(struct config_enum *record,
  									  const char *value, int *retval);
  
+ extern void DefineCustomVariable(enum config_type type, const void *variable);
  
  #endif   /* GUC_TABLES_H */
#51Jeff Davis
pgsql@j-davis.com
In reply to: ITAGAKI Takahiro (#50)
1 attachment(s)
Re: auto_explain contrib moudle

On Mon, 2008-11-10 at 18:02 +0900, ITAGAKI Takahiro wrote:

That's because explain.log_analyze requires executor instruments,
and it's not free. I think we'd better to have an option to avoid
the overheads... Oops, I found my bug when force_instrument is
turned on. It should be enabled only when
(explain_log_min_duration >= 0 && explain_log_analyze).

I send a new patch to fix it. A documentation about
explain.log_nested_statements is also added to the sgml file.

Great. I attached a patch with some minor documentation changes.

There seems to be no auto_explain view, so I assumed that section of the
docs was old, and I removed it. Let me know if you intend to include the
view.

Thanks! This patch is ready to go, as far as I'm concerned.

Regards,
Jeff Davis

Attachments:

auto-explain.1111.patchtext/x-patch; charset=utf-8; name=auto-explain.1111.patchDownload
diff --git a/contrib/Makefile b/contrib/Makefile
index 30f75c7..b25af3c 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -6,6 +6,7 @@ include $(top_builddir)/src/Makefile.global
 
 WANTED_DIRS = \
 		adminpack	\
+		auto_explain	\
 		btree_gist	\
 		chkpass		\
 		citext		\
diff --git a/contrib/auto_explain/Makefile b/contrib/auto_explain/Makefile
new file mode 100644
index 0000000..0d0ccc2
--- /dev/null
+++ b/contrib/auto_explain/Makefile
@@ -0,0 +1,13 @@
+MODULE_big = auto_explain
+OBJS = auto_explain.o
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/auto_explain
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
new file mode 100644
index 0000000..af994e7
--- /dev/null
+++ b/contrib/auto_explain/auto_explain.c
@@ -0,0 +1,215 @@
+/*
+ * auto_explain.c
+ */
+#include "postgres.h"
+
+#include "commands/explain.h"
+#include "executor/instrument.h"
+#include "miscadmin.h"
+#include "utils/guc_tables.h"
+
+PG_MODULE_MAGIC;
+
+#define GUCNAME(name)		("explain." name)
+
+static int		explain_log_min_duration = -1;	/* msec or -1 */
+static bool		explain_log_analyze = false;
+static bool		explain_log_verbose = false;
+static bool		explain_log_nested = false;
+
+static bool		toplevel = true;
+static ExecutorRun_hook_type	prev_ExecutorRun = NULL;
+
+void	_PG_init(void);
+void	_PG_fini(void);
+
+static void explain_ExecutorRun(QueryDesc *queryDesc,
+								ScanDirection direction,
+								long count);
+static bool assign_log_min_duration(int newval, bool doit, GucSource source);
+static bool assign_log_analyze(bool newval, bool doit, GucSource source);
+static bool assign_log_verbose(bool newval, bool doit, GucSource source);
+static bool assign_log_nested(bool newval, bool doit, GucSource source);
+
+static struct config_int def_log_min_duration =
+{
+	{
+		GUCNAME("log_min_duration"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Sets the minimum execution time above which plans will be logged.",
+		"Zero prints all plans. -1 turns this feature off.",
+		GUC_UNIT_MS
+	},
+	&explain_log_min_duration,
+	-1, -1, INT_MAX / 1000, assign_log_min_duration, NULL
+};
+
+static struct config_bool def_log_analyze =
+{
+	{
+		GUCNAME("log_analyze"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN ANALYZE for plan logging."
+	},
+	&explain_log_analyze,
+	false, assign_log_analyze, NULL
+};
+
+static struct config_bool def_log_verbose =
+{
+	{
+		GUCNAME("log_verbose"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Use EXPLAIN VERBOSE for plan logging."
+	},
+	&explain_log_verbose,
+	false, assign_log_verbose, NULL
+};
+
+static struct config_bool def_log_nested_statements =
+{
+	{
+		GUCNAME("log_nested_statements"),
+		PGC_USERSET,
+		STATS_MONITORING,
+		"Log nested statements."
+	},
+	&explain_log_nested,
+	false, assign_log_nested, NULL
+};
+
+void
+_PG_init(void)
+{
+	DefineCustomVariable(PGC_INT, &def_log_min_duration);
+	DefineCustomVariable(PGC_BOOL, &def_log_analyze);
+	DefineCustomVariable(PGC_BOOL, &def_log_verbose);
+	DefineCustomVariable(PGC_BOOL, &def_log_nested_statements);
+
+	/* install ExecutorRun_hook */
+	prev_ExecutorRun = ExecutorRun_hook;
+	ExecutorRun_hook = explain_ExecutorRun;
+}
+
+void
+_PG_fini(void)
+{
+	/* uninstall ExecutorRun_hook */
+	ExecutorRun_hook = prev_ExecutorRun;
+}
+
+void
+explain_ExecutorRun(QueryDesc *queryDesc, ScanDirection direction, long count)
+{
+	if ((toplevel || explain_log_nested) && explain_log_min_duration >= 0)
+	{
+		instr_time		starttime;
+		instr_time		duration;
+		double			msec;
+
+		/* Disable our hooks temporarily during the top-level query. */
+		toplevel = false;
+		PG_TRY();
+		{
+			INSTR_TIME_SET_CURRENT(starttime);
+
+			if (prev_ExecutorRun)
+				prev_ExecutorRun(queryDesc, direction, count);
+			else
+				standard_ExecutorRun(queryDesc, direction, count);
+
+			INSTR_TIME_SET_CURRENT(duration);
+			INSTR_TIME_SUBTRACT(duration, starttime);
+			msec = INSTR_TIME_GET_MILLISEC(duration);
+
+			/* Log plan if duration is exceeded. */
+			if (msec > explain_log_min_duration)
+			{
+				StringInfoData	buf;
+
+				initStringInfo(&buf);
+				ExplainOneResult(&buf, queryDesc,
+					queryDesc->doInstrument && explain_log_analyze,
+					explain_log_verbose);
+
+				/* Remove last line break */
+				if (buf.len > 0 && buf.data[buf.len - 1] == '\n')
+				{
+					buf.data[buf.len - 1] = '\0';
+					buf.len--;
+				}
+				ereport(LOG,
+						(errmsg("duration: %.3f ms  plan:\n%s",
+								msec, buf.data)));
+
+				pfree(buf.data);
+			}
+
+			toplevel = true;
+		}
+		PG_CATCH();
+		{
+			toplevel = true;
+			PG_RE_THROW();
+		}
+		PG_END_TRY();
+	}
+	else
+	{
+		/* ignore recursive executions, that are typically function calls */
+		if (prev_ExecutorRun)
+			prev_ExecutorRun(queryDesc, direction, count);
+		else
+			standard_ExecutorRun(queryDesc, direction, count);
+	}
+}
+
+/* Emulate PGC_SUSET for custom variables. */
+static bool
+suset_assign(GucSource source, const char *name)
+{
+	if (source >= PGC_S_CLIENT && !superuser())
+	{
+		ereport(GUC_complaint_elevel(source),
+				(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+				 errmsg("permission denied to set parameter \"%s\"", name)));
+		return false;
+	}
+
+	return true;
+}
+
+static bool
+assign_log_min_duration(int newval, bool doit, GucSource source)
+{
+	if (!suset_assign(source, GUCNAME("log_min_duration")))
+		return false;
+	if (doit)
+		force_instrument = (newval >= 0 && explain_log_analyze);
+	return true;
+}
+
+static bool
+assign_log_analyze(bool newval, bool doit, GucSource source)
+{
+	if (!suset_assign(source, GUCNAME("log_analyze")))
+		return false;
+	if (doit)
+		force_instrument = (explain_log_min_duration >= 0 && newval);
+	return true;
+}
+
+static bool
+assign_log_verbose(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_verbose"));
+}
+
+static bool
+assign_log_nested(bool newval, bool doit, GucSource source)
+{
+	return suset_assign(source, GUCNAME("log_nested_statements"));
+}
diff --git a/doc/src/sgml/auto-explain.sgml b/doc/src/sgml/auto-explain.sgml
new file mode 100644
index 0000000..94b7daf
--- /dev/null
+++ b/doc/src/sgml/auto-explain.sgml
@@ -0,0 +1,141 @@
+<sect1 id="autoexplain">
+ <title>auto_explain</title>
+
+ <indexterm zone="autoexplain">
+  <primary>auto_explain</primary>
+ </indexterm>
+
+ <para>
+  The <filename>auto_explain</filename> module provides a means for
+  logging execution plans automatically, without having to run <command>EXPLAIN</>.
+ </para>
+
+ <para>
+  You can <command>LOAD</> this module dynamically or preload it automatically with
+  <varname>shared_preload_libraries</> or <varname>local_preload_libraries</>.
+ </para>
+
+ <sect2>
+  <title>Configuration parameters</title>
+
+  <variablelist>
+   <varlistentry>
+    <term>
+     <varname>explain.log_min_duration</varname> (<type>integer</type>)
+    </term>
+    <listitem>
+     <para>
+      <varname>explain.log_min_duration</varname> is the minimum execution time
+      in milliseconds that will cause the plan to be logged. Setting this to zero
+      logs all plans. Minus-one (the default) disables logging of plans.
+      For example, if you set it to <literal>250ms</literal> then all plans
+      that run 250ms or longer in executor will be logged.
+      Enabling this parameter can be helpful in tracking down unoptimized queries
+      in your applications. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_analyze</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_analyze</varname> causes <command>EXPLAIN ANALYZE</>
+      to be used rather than normal <command>EXPLAIN</> when an execution plan
+      is logged. This parameter is off by default. Only superusers can change
+      this setting.
+     </para>
+     <para>
+      NOTE: If you set the parameter on, instrument timers are enabled even if
+      you don't use EXPLAIN ANALYZE. This creates some overhead for all
+      statements executed.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_verbose</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_verbose</varname> causes <command>EXPLAIN VERBOSE</>
+      to be used rather than normal <command>EXPLAIN</> when an execution plan
+      is logged. This parameter is off by default. Only superusers can change
+      this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
+    <term>
+     <varname>explain.log_nested_statements</varname> (<type>boolean</type>)
+    </term>
+
+    <listitem>
+     <para>
+      <varname>explain.log_nested_statements</varname> causes the execution plans
+      of nested statements (statements executed inside a function) to be logged
+      as well. If the value is off, only top-level plans are logged. This
+      parameter is off by default. Only superusers can change this setting.
+     </para>
+    </listitem>
+   </varlistentry>
+  </variablelist>
+
+  <para>
+   If you set these explain.* parameters in your postgresql.conf,
+   you also need to add 'explain' in <varname>custom_variable_classes</>.
+  </para>
+
+  <programlisting>
+  # postgresql.conf
+  shared_preload_libraries = 'auto_explain'
+  
+  custom_variable_classes = 'explain'
+  explain.log_min_duration = 3s
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Example</title>
+
+  <programlisting>
+  postgres=# LOAD 'auto_explain';
+  postgres=# SET explain.log_min_duration = 0;
+  postgres=# SELECT count(*)
+               FROM pg_class, pg_index
+              WHERE oid = indrelid AND indisunique;
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Sample Log Output</title>
+  <programlisting>
+  LOG:  duration: 0.986 ms  plan:
+          Aggregate  (cost=14.90..14.91 rows=1 width=0)
+            ->  Hash Join  (cost=3.91..14.70 rows=81 width=0)
+                  Hash Cond: (pg_class.oid = pg_index.indrelid)
+                  ->  Seq Scan on pg_class  (cost=0.00..8.27 rows=227 width=4)
+                  ->  Hash  (cost=2.90..2.90 rows=81 width=4)
+                        ->  Seq Scan on pg_index  (cost=0.00..2.90 rows=81 width=4)
+                              Filter: indisunique
+  STATEMENT:  SELECT count(*)
+            FROM pg_class, pg_index
+           WHERE oid = indrelid AND indisunique;
+  </programlisting>
+ </sect2>
+
+ <sect2>
+  <title>Authors</title>
+
+  <para>
+   Takahiro Itagaki <email>itagaki.takahiro@oss.ntt.co.jp</email>
+  </para>
+ </sect2>
+
+</sect1>
diff --git a/doc/src/sgml/contrib.sgml b/doc/src/sgml/contrib.sgml
index 72d8828..0108da3 100644
--- a/doc/src/sgml/contrib.sgml
+++ b/doc/src/sgml/contrib.sgml
@@ -79,6 +79,7 @@ psql -d dbname -f <replaceable>SHAREDIR</>/contrib/<replaceable>module</>.sql
  </para>
 
  &adminpack;
+ &auto-explain;
  &btree-gist;
  &chkpass;
  &citext;
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index b0538b6..68db2b9 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -92,6 +92,7 @@
 <!-- contrib information -->
 <!entity contrib         SYSTEM "contrib.sgml">
 <!entity adminpack       SYSTEM "adminpack.sgml">
+<!entity auto-explain    SYSTEM "auto-explain.sgml">
 <!entity btree-gist      SYSTEM "btree-gist.sgml">
 <!entity chkpass         SYSTEM "chkpass.sgml">
 <!entity citext          SYSTEM "citext.sgml">
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index c0c53cf..4d46a34 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -224,7 +224,6 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	QueryDesc  *queryDesc;
 	instr_time	starttime;
 	double		totaltime = 0;
-	ExplainState *es;
 	StringInfoData buf;
 	int			eflags;
 
@@ -265,17 +264,8 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 		totaltime += elapsed_time(&starttime);
 	}
 
-	es = (ExplainState *) palloc0(sizeof(ExplainState));
-
-	es->printTList = stmt->verbose;
-	es->printAnalyze = stmt->analyze;
-	es->pstmt = queryDesc->plannedstmt;
-	es->rtable = queryDesc->plannedstmt->rtable;
-
 	initStringInfo(&buf);
-	explain_outNode(&buf,
-					queryDesc->plannedstmt->planTree, queryDesc->planstate,
-					NULL, 0, es);
+	ExplainOneResult(&buf, queryDesc, stmt->analyze, stmt->verbose);
 
 	/*
 	 * If we ran the command, run any AFTER triggers it queued.  (Note this
@@ -290,7 +280,7 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	}
 
 	/* Print info about runtime of triggers */
-	if (es->printAnalyze)
+	if (stmt->analyze)
 	{
 		ResultRelInfo *rInfo;
 		bool		show_relname;
@@ -335,7 +325,26 @@ ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 	do_text_output_multiline(tstate, buf.data);
 
 	pfree(buf.data);
-	pfree(es);
+}
+
+/*
+ * ExplainOneResult -
+ *	  converts a Plan node into ascii string and appends it to 'str'
+ */
+void
+ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+				 bool analyze, bool verbose)
+{
+	ExplainState	es = { 0 };
+
+	es.printTList = verbose;
+	es.printAnalyze = analyze;
+	es.pstmt = queryDesc->plannedstmt;
+	es.rtable = queryDesc->plannedstmt->rtable;
+
+	explain_outNode(str,
+					queryDesc->plannedstmt->planTree, queryDesc->planstate,
+					NULL, 0, &es);
 }
 
 /*
diff --git a/src/backend/tcop/pquery.c b/src/backend/tcop/pquery.c
index a933d3c..ff2df9b 100644
--- a/src/backend/tcop/pquery.c
+++ b/src/backend/tcop/pquery.c
@@ -32,7 +32,7 @@
  * if there are several).
  */
 Portal		ActivePortal = NULL;
-
+bool		force_instrument = false;
 
 static void ProcessQuery(PlannedStmt *plan,
 			 ParamListInfo params,
@@ -76,7 +76,7 @@ CreateQueryDesc(PlannedStmt *plannedstmt,
 	qd->crosscheck_snapshot = RegisterSnapshot(crosscheck_snapshot);
 	qd->dest = dest;			/* output dest */
 	qd->params = params;		/* parameter values passed into query */
-	qd->doInstrument = doInstrument;	/* instrumentation wanted? */
+	qd->doInstrument = force_instrument || doInstrument;	/* instrumentation wanted? */
 
 	/* null these fields until set by ExecutorStart */
 	qd->tupDesc = NULL;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fd7543b..f8a7e64 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -169,6 +169,7 @@ static const char *assign_pgstat_temp_directory(const char *newval, bool doit, G
 
 static char *config_enum_get_options(struct config_enum *record, 
 									 const char *prefix, const char *suffix);
+static void initialize_option(struct config_generic *gconf);
 
 
 /*
@@ -3194,112 +3195,7 @@ InitializeGUCOptions(void)
 	for (i = 0; i < num_guc_variables; i++)
 	{
 		struct config_generic *gconf = guc_variables[i];
-
-		gconf->status = 0;
-		gconf->reset_source = PGC_S_DEFAULT;
-		gconf->source = PGC_S_DEFAULT;
-		gconf->stack = NULL;
-		gconf->sourcefile = NULL;
-		gconf->sourceline = 0;
-
-		switch (gconf->vartype)
-		{
-			case PGC_BOOL:
-				{
-					struct config_bool *conf = (struct config_bool *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, (int) conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_INT:
-				{
-					struct config_int *conf = (struct config_int *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %d",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_REAL:
-				{
-					struct config_real *conf = (struct config_real *) gconf;
-
-					Assert(conf->boot_val >= conf->min);
-					Assert(conf->boot_val <= conf->max);
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %g",
-								 conf->gen.name, conf->boot_val);
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-			case PGC_STRING:
-				{
-					struct config_string *conf = (struct config_string *) gconf;
-					char	   *str;
-
-					*conf->variable = NULL;
-					conf->reset_val = NULL;
-
-					if (conf->boot_val == NULL)
-					{
-						/* leave the value NULL, do not call assign hook */
-						break;
-					}
-
-					str = guc_strdup(FATAL, conf->boot_val);
-					conf->reset_val = str;
-
-					if (conf->assign_hook)
-					{
-						const char *newstr;
-
-						newstr = (*conf->assign_hook) (str, true,
-													   PGC_S_DEFAULT);
-						if (newstr == NULL)
-						{
-							elog(FATAL, "failed to initialize %s to \"%s\"",
-								 conf->gen.name, str);
-						}
-						else if (newstr != str)
-						{
-							free(str);
-
-							/*
-							 * See notes in set_config_option about casting
-							 */
-							str = (char *) newstr;
-							conf->reset_val = str;
-						}
-					}
-					*conf->variable = str;
-					break;
-				}
-			case PGC_ENUM:
-				{
-					struct config_enum *conf = (struct config_enum *) gconf;
-
-					if (conf->assign_hook)
-						if (!(*conf->assign_hook) (conf->boot_val, true,
-												   PGC_S_DEFAULT))
-							elog(FATAL, "failed to initialize %s to %s",
-								 conf->gen.name, 
-								 config_enum_lookup_by_value(conf, conf->boot_val));
-					*conf->variable = conf->reset_val = conf->boot_val;
-					break;
-				}
-		}
+		initialize_option(gconf);
 	}
 
 	guc_dirty = false;
@@ -3355,6 +3251,115 @@ InitializeGUCOptions(void)
 	}
 }
 
+static void
+initialize_option(struct config_generic *gconf)
+{
+	gconf->status = 0;
+	gconf->reset_source = PGC_S_DEFAULT;
+	gconf->source = PGC_S_DEFAULT;
+	gconf->stack = NULL;
+	gconf->sourcefile = NULL;
+	gconf->sourceline = 0;
+
+	switch (gconf->vartype)
+	{
+		case PGC_BOOL:
+			{
+				struct config_bool *conf = (struct config_bool *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, (int) conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_INT:
+			{
+				struct config_int *conf = (struct config_int *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %d",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_REAL:
+			{
+				struct config_real *conf = (struct config_real *) gconf;
+
+				Assert(conf->boot_val >= conf->min);
+				Assert(conf->boot_val <= conf->max);
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %g",
+							 conf->gen.name, conf->boot_val);
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+		case PGC_STRING:
+			{
+				struct config_string *conf = (struct config_string *) gconf;
+				char	   *str;
+
+				*conf->variable = NULL;
+				conf->reset_val = NULL;
+
+				if (conf->boot_val == NULL)
+				{
+					/* leave the value NULL, do not call assign hook */
+					break;
+				}
+
+				str = guc_strdup(FATAL, conf->boot_val);
+				conf->reset_val = str;
+
+				if (conf->assign_hook)
+				{
+					const char *newstr;
+
+					newstr = (*conf->assign_hook) (str, true,
+												   PGC_S_DEFAULT);
+					if (newstr == NULL)
+					{
+						elog(FATAL, "failed to initialize %s to \"%s\"",
+							 conf->gen.name, str);
+					}
+					else if (newstr != str)
+					{
+						free(str);
+
+						/*
+						 * See notes in set_config_option about casting
+						 */
+						str = (char *) newstr;
+						conf->reset_val = str;
+					}
+				}
+				*conf->variable = str;
+				break;
+			}
+		case PGC_ENUM:
+			{
+				struct config_enum *conf = (struct config_enum *) gconf;
+
+				if (conf->assign_hook)
+					if (!(*conf->assign_hook) (conf->boot_val, true,
+											   PGC_S_DEFAULT))
+						elog(FATAL, "failed to initialize %s to %s",
+							 conf->gen.name, 
+							 config_enum_lookup_by_value(conf, conf->boot_val));
+				*conf->variable = conf->reset_val = conf->boot_val;
+				break;
+			}
+	}
+}
 
 /*
  * Select the configuration files and data directory to be used, and
@@ -5656,6 +5661,7 @@ define_custom_variable(struct config_generic * variable)
 	if (res == NULL)
 	{
 		/* No placeholder to replace, so just add it */
+		initialize_option(variable);
 		add_guc_variable(variable, ERROR);
 		return;
 	}
@@ -5690,6 +5696,8 @@ define_custom_variable(struct config_generic * variable)
 		set_config_option(name, value,
 						  pHolder->gen.context, pHolder->gen.source,
 						  GUC_ACTION_SET, true);
+	else
+		initialize_option(variable);
 
 	/*
 	 * Free up as much as we conveniently can of the placeholder structure
@@ -5823,6 +5831,29 @@ DefineCustomEnumVariable(const char *name,
 	define_custom_variable(&var->gen);
 }
 
+static const int config_varsize[] =
+{
+	sizeof(struct config_bool),
+	sizeof(struct config_int),
+	sizeof(struct config_real),
+	sizeof(struct config_string),
+	sizeof(struct config_enum),
+};
+
+void
+DefineCustomVariable(enum config_type type, const void *variable)
+{
+	int		size = config_varsize[type];
+	const struct config_generic	   *var = variable;
+	struct config_generic		   *gen;
+
+	gen = (struct config_generic *) guc_malloc(ERROR, size);
+	memcpy(gen, var, size);
+	gen->name = guc_strdup(ERROR, var->name);
+	gen->vartype = type;
+	define_custom_variable(gen);
+}
+
 void
 EmitWarningsOnPlaceholders(const char *className)
 {
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 5b808d5..cf07915 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -41,4 +41,7 @@ extern void ExplainOneUtility(Node *utilityStmt, ExplainStmt *stmt,
 extern void ExplainOnePlan(PlannedStmt *plannedstmt, ParamListInfo params,
 			   ExplainStmt *stmt, TupOutputState *tstate);
 
+extern void ExplainOneResult(StringInfo str, QueryDesc *queryDesc,
+							 bool analyze, bool verbose);
+
 #endif   /* EXPLAIN_H */
diff --git a/src/include/executor/execdesc.h b/src/include/executor/execdesc.h
index 2e2c159..854721c 100644
--- a/src/include/executor/execdesc.h
+++ b/src/include/executor/execdesc.h
@@ -20,6 +20,8 @@
 #include "tcop/dest.h"
 
 
+extern PGDLLIMPORT bool force_instrument;
+
 /* ----------------
  *		query descriptor:
  *
diff --git a/src/include/utils/guc_tables.h b/src/include/utils/guc_tables.h
index 8791660..3982a27 100644
--- a/src/include/utils/guc_tables.h
+++ b/src/include/utils/guc_tables.h
@@ -242,5 +242,6 @@ extern const char *config_enum_lookup_by_value(struct config_enum *record, int v
 extern bool config_enum_lookup_by_name(struct config_enum *record,
 									  const char *value, int *retval);
 
+extern void DefineCustomVariable(enum config_type type, const void *variable);
 
 #endif   /* GUC_TABLES_H */
#52Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Davis (#51)
Re: auto_explain contrib moudle

Jeff Davis <pgsql@j-davis.com> writes:

Thanks! This patch is ready to go, as far as I'm concerned.

This patch seems to contain a subset of the "contrib infrastructure"
patch that's listed separately on the commitfest page. While I have
no strong objection to what's here, I'm wondering what sort of process
we want to follow. Is the infrastructure stuff getting separately
reviewed or not?

regards, tom lane

#53Jeff Davis
pgsql@j-davis.com
In reply to: Tom Lane (#52)
Re: auto_explain contrib moudle

On Thu, 2008-11-13 at 14:31 -0500, Tom Lane wrote:

Jeff Davis <pgsql@j-davis.com> writes:

Thanks! This patch is ready to go, as far as I'm concerned.

This patch seems to contain a subset of the "contrib infrastructure"
patch that's listed separately on the commitfest page. While I have
no strong objection to what's here, I'm wondering what sort of process
we want to follow. Is the infrastructure stuff getting separately
reviewed or not?

I can review it, but not until this weekend. It looks like someone
already added me to the list of reviewers on that patch. I'm not sure if
Matthew Wetmore has already started reviewing it or not.

Regards,
Jeff Davis

#54Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Davis (#53)
Re: auto_explain contrib moudle

Jeff Davis <pgsql@j-davis.com> writes:

On Thu, 2008-11-13 at 14:31 -0500, Tom Lane wrote:

This patch seems to contain a subset of the "contrib infrastructure"
patch that's listed separately on the commitfest page. While I have
no strong objection to what's here, I'm wondering what sort of process
we want to follow. Is the infrastructure stuff getting separately
reviewed or not?

I can review it, but not until this weekend. It looks like someone
already added me to the list of reviewers on that patch. I'm not sure if
Matthew Wetmore has already started reviewing it or not.

Now that I look closer, the "contrib infrastructure" item is just a
combination of the auto_explain and pg_stat_statements items, and I
guess the reason you and Matthew were shown as reviewers was that
you'd each been assigned one of those two items. As far as I can see
this is just confusing and duplicative. I've removed the
"infrastructure" item from the commitfest page; I think we can
proceed with the two other items separately. If there's any conflict
in the two patches we can resolve it after the first one gets applied.

regards, tom lane

#55ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Tom Lane (#54)
Re: auto_explain contrib moudle

Tom Lane <tgl@sss.pgh.pa.us> wrote:

Now that I look closer, the "contrib infrastructure" item is just a
combination of the auto_explain and pg_stat_statements items, and I
guess the reason you and Matthew were shown as reviewers was that
you'd each been assigned one of those two items. As far as I can see
this is just confusing and duplicative.

That's right. Sorry for your confusing.

I think we can
proceed with the two other items separately. If there's any conflict
in the two patches we can resolve it after the first one gets applied.

contrib/auto_explain patch is "Ready for committer" even without
the QueryDesc patch. So please apply it if there are no problems.
I'll rewrite contrib/pg_stat_statements to use the new feature
in the QueryDesc patch and avoid confliction from the first one.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#56Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Davis (#51)
Re: auto_explain contrib moudle

Jeff Davis <pgsql@j-davis.com> writes:

On Mon, 2008-11-10 at 18:02 +0900, ITAGAKI Takahiro wrote:

That's because explain.log_analyze requires executor instruments,
and it's not free. I think we'd better to have an option to avoid
the overheads... Oops, I found my bug when force_instrument is
turned on. It should be enabled only when
(explain_log_min_duration >= 0 && explain_log_analyze).

I send a new patch to fix it. A documentation about
explain.log_nested_statements is also added to the sgml file.

Great. I attached a patch with some minor documentation changes.

Looking at this patch now ...

I don't like the force_instrument thing too much at all.  It's brute
force and it doesn't get every case right today, much less in the future
--- for instance, it uselessly forces instrumentation on an EXPLAIN
startup, because it can't react to EXEC_FLAG_EXPLAIN_ONLY.

I think a cleaner solution here is to create a hook for ExecutorStart
just as we have done for ExecutorRun --- and probably there ought to
be one for ExecutorEnd for completeness. auto_explain would then
hook into ExecutorStart and force doInstrument true on appropriate
conditions.

Another issue that is bothering me is that it's not clear that an
ExecutorRun execution is the right unit for measuring the runtime of a
query --- this would be quite misleading for queries that are executed a
bit at a time, such as SQL functions and cursor queries. I think it'd
be better to accumulate runtime and then report in an ExecutorEnd hook.
This would likely require adding a struct Instrumentation * field to
QueryDesc in which to track the total ExecutorRun timing, but I don't
see anything very wrong with that. The core system would just leave it
NULL, and the ExecutorStart hook could fill it in when it wants the
query to be tracked by the other two hooks.

regards, tom lane

#57ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Tom Lane (#56)
Re: auto_explain contrib moudle

Ok, I'm looking at the direction of ExecutorStart/End hooks...

Tom Lane <tgl@sss.pgh.pa.us> wrote:

This would likely require adding a struct Instrumentation * field to
QueryDesc in which to track the total ExecutorRun timing

I think instr_time is enough here,
but why do you think Instrumentation is needed?

There might be another approach to have a stopwatch stack in
the contrib module instead of the core. I think it is cleaner
because it works even if multiple modules use the stopwatch
at the same time. Am I missing something?

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#58ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: ITAGAKI Takahiro (#57)
Re: auto_explain contrib moudle

I wrote:

There might be another approach to have a stopwatch stack in
the contrib module instead of the core. I think it is cleaner
because it works even if multiple modules use the stopwatch
at the same time. Am I missing something?

Ooops, it should be:
... because each multiple module can use its own stopwatch
stack at the same time.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

#59Tom Lane
tgl@sss.pgh.pa.us
In reply to: ITAGAKI Takahiro (#57)
Re: auto_explain contrib moudle

ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:

Tom Lane <tgl@sss.pgh.pa.us> wrote:

This would likely require adding a struct Instrumentation * field to
QueryDesc in which to track the total ExecutorRun timing

I think instr_time is enough here,
but why do you think Instrumentation is needed?

Well, it's more general. You could think of ExecutorRun as being like a
plan node, in which case all the fields of struct Instrumentation are
potentially useful. If a contrib module doesn't want to use them,
of course it doesn't have to.

Or I guess we could just put a void * there and not try to wire down
its usage.

There might be another approach to have a stopwatch stack in
the contrib module instead of the core. I think it is cleaner
because it works even if multiple modules use the stopwatch
at the same time. Am I missing something?

I think this will be mighty messy ...

regards, tom lane

#60ITAGAKI Takahiro
itagaki.takahiro@oss.ntt.co.jp
In reply to: Tom Lane (#59)
1 attachment(s)
Re: auto_explain contrib moudle

Here is an update version of contrib/auto_explain patch.

Now it uses new ExecutorStart_hook and ExecutorEnd_hook.
When we execute queries using cursor, FETCHes are accumulated
and reported only once on CLOSE.

A new argument 'flags' is added in DefineCustomXXXVariable()
and custom GUC variables are defined with it. Some GUC_* constants
are moved to guc.h from guc_tables.h.

A variable "Instrumentation *totaltime" is added in QueryDesc.
It is not used by the core in default (always NULL), but initialized
by contrib modules. However, the core update the counter if it is
not NULL because multiple contrib modules might use it.
If the core don't update the counter, only one of the modules
should update it, but it's hard to determine which one should do.

Tom Lane <tgl@sss.pgh.pa.us> wrote:

This would likely require adding a struct Instrumentation * field to

Well, it's more general. You could think of ExecutorRun as being like a
plan node, in which case all the fields of struct Instrumentation are
potentially useful. If a contrib module doesn't want to use them,
of course it doesn't have to.

Instrumentation is enough for auto_explain, so I used it for now.
However, pg_stat_statements also use the field and requires other
counters (buffer usages and rusage). It might replace the field
with another struct. Then we might need to reconsider how to deal
with session statistics, suggested by Vladimir.

"Vladimir Sitnikov" <sitnikov.vladimir@gmail.com> wrote:

I wish PostgreSQL had some kind of pg_session_statistics view that reports
resource usage statistics for each session.
For instance, it could expose "buffer usage" to the client, so it could get
more details on resource usage. For instance, I would like to see a new tab
in pgAdmin that shows "total number of buffer gets", "number of WAL records
created", "number of rows sorted" and similar information after query
finishes (even in plain "execute" mode).

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Attachments:

auto_explain.1118.tar.gzapplication/octet-stream; name=auto_explain.1118.tar.gzDownload
#61Tom Lane
tgl@sss.pgh.pa.us
In reply to: ITAGAKI Takahiro (#60)
Re: auto_explain contrib moudle

ITAGAKI Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:

Here is an update version of contrib/auto_explain patch.

Applied with some editorialization, mostly around the GUC stuff.

regards, tom lane