top-level DML under CTEs

Started by Hitoshi Haradaover 15 years ago29 messages
#1Hitoshi Harada
umi.tanuki@gmail.com
1 attachment(s)

The patch attached is based on the one rejected at the last CF for 9.0
last year.

http://archives.postgresql.org/message-id/16303.1266023203@sss.pgh.pa.us

This patch implements the feature that allows top-level DMLs under CTE
WITH clause. For example:

WITH t AS (SELECT * FROM x)
UPDATE y SET val = t.val FROM t
WHERE y.key = t.key;

This feature is part of writeable CTEs proposed by David Fetter originally.

There were two issues at the CF.

1. WITH clause atop INSERT
Although the previous discussion got the consensus that we forbid WITH
atop INSERT, it seems to me that it can be allowed. I managed to do it
by treating the top WITH clause (of INSERT) as if the one of SELECT
(or VALUES). It is possible to disallow the CTE over INSERT statement,
but the lack for INSERT, though there are for UPDATE and DELETE,
sounds inconsistent enough.

2. OLD/NEW in rules
Following the subsequent discussion after the post linked above, I add
code to throw an appropriate error when OLD/NEW is used in WITH
clauses. It is true that OLD/NEW references look sane to general
users, but actually (at least in our implementation) they are located
in the top-level query's Range Table List. Consequently, they are
invisible inside the WITH clause. To allow them, we should rewrite the
rule systems overall. Thus, we forbid them in WITH though we should
throw an error indicating appropriate message.

I'll add the entry to CF app later. Any feedback is welcome.

Regards,

--
Hitoshi Harada

Attachments:

toplevel-dml-cte.20100913.patchapplication/octet-stream; name=toplevel-dml-cte.20100913.patchDownload
*** a/doc/src/sgml/ref/delete.sgml
--- b/doc/src/sgml/ref/delete.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
      [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
***************
*** 84,89 **** DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
--- 85,102 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><literal>ONLY</></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/insert.sgml
--- b/doc/src/sgml/ref/insert.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
      { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
      [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
***************
*** 85,90 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
--- 86,109 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+      <para>
+       It is possible that <literal>SELECT</literal> query also has
+       <literal>WITH</literal>.  In this case the two
+       <replaceable>with_query</replaceable> can be referred from
+       the <literal>SELECT</literal> query.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/update.sgml
--- b/doc/src/sgml/ref/update.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
            ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
***************
*** 80,85 **** UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
--- 81,98 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***************
*** 2287,2292 **** _copyInsertStmt(InsertStmt *from)
--- 2287,2293 ----
  	COPY_NODE_FIELD(cols);
  	COPY_NODE_FIELD(selectStmt);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2300,2305 **** _copyDeleteStmt(DeleteStmt *from)
--- 2301,2307 ----
  	COPY_NODE_FIELD(usingClause);
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2314,2319 **** _copyUpdateStmt(UpdateStmt *from)
--- 2316,2322 ----
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(fromClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
*** a/src/backend/nodes/equalfuncs.c
--- b/src/backend/nodes/equalfuncs.c
***************
*** 889,894 **** _equalInsertStmt(InsertStmt *a, InsertStmt *b)
--- 889,895 ----
  	COMPARE_NODE_FIELD(cols);
  	COMPARE_NODE_FIELD(selectStmt);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 900,905 **** _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
--- 901,907 ----
  	COMPARE_NODE_FIELD(usingClause);
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 912,917 **** _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
--- 914,920 ----
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(fromClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
*** a/src/backend/parser/analyze.c
--- b/src/backend/parser/analyze.c
***************
*** 282,287 **** transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
--- 282,294 ----
  
  	qry->commandType = CMD_DELETE;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/* set up range table with just the result rel */
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
***************
*** 343,348 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
--- 350,374 ----
  	pstate->p_is_insert = true;
  
  	/*
+ 	 * The WITH of INSERT is interpreted as WITH of SELECT (or VALUES) and
+ 	 * the original WITH of SELECT is appended to that of INSERT.
+ 	 */
+ 	if (stmt->withClause)
+ 	{
+ 		if (selectStmt && selectStmt->withClause)
+ 		{
+ 			WithClause *with = selectStmt->withClause;
+ 			with->recursive |= stmt->withClause->recursive;
+ 			with->ctes = list_concat(copyObject(stmt->withClause->ctes), with->ctes);
+ 		}
+ 		else if (selectStmt)
+ 		{
+ 			selectStmt->withClause = stmt->withClause;
+ 		}
+ 		stmt->withClause = NULL;
+ 	}
+ 
+ 	/*
  	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
  	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
  	 * efficiency and so we can handle DEFAULT specifications.
***************
*** 1726,1731 **** transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
--- 1752,1764 ----
  	qry->commandType = CMD_UPDATE;
  	pstate->p_is_update = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
  										 true,
*** a/src/backend/parser/gram.y
--- b/src/backend/parser/gram.y
***************
*** 430,436 **** static TypeName *TableFuncTypeName(List *columns);
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
--- 430,436 ----
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause opt_with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
***************
*** 7129,7139 **** DeallocateStmt: DEALLOCATE name
   *****************************************************************************/
  
  InsertStmt:
! 			INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$4->relation = $3;
! 					$4->returningList = $5;
! 					$$ = (Node *) $4;
  				}
  		;
  
--- 7129,7140 ----
   *****************************************************************************/
  
  InsertStmt:
! 			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$5->relation = $4;
! 					$5->returningList = $6;
! 					$5->withClause = $1;
! 					$$ = (Node *) $5;
  				}
  		;
  
***************
*** 7189,7202 **** returning_clause:
   *
   *****************************************************************************/
  
! DeleteStmt: DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $3;
! 					n->usingClause = $4;
! 					n->whereClause = $5;
! 					n->returningList = $6;
  					$$ = (Node *)n;
  				}
  		;
--- 7190,7204 ----
   *
   *****************************************************************************/
  
! DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $4;
! 					n->usingClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7251,7268 **** opt_nowait:	NOWAIT							{ $$ = TRUE; }
   *
   *****************************************************************************/
  
! UpdateStmt: UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $2;
! 					n->targetList = $4;
! 					n->fromClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
  					$$ = (Node *)n;
  				}
  		;
--- 7253,7271 ----
   *
   *****************************************************************************/
  
! UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $3;
! 					n->targetList = $5;
! 					n->fromClause = $6;
! 					n->whereClause = $7;
! 					n->returningList = $8;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7604,7609 **** common_table_expr:  name opt_name_list AS select_with_parens
--- 7607,7618 ----
  			}
  		;
  
+ opt_with_clause:
+ 		with_clause								{ $$ = $1; }
+ 		| /*EMPTY*/								{ $$ = NULL; }
+ 		;
+ 
+ 
  into_clause:
  			INTO OptTempTableName
  				{
*** a/src/backend/parser/parse_utilcmd.c
--- b/src/backend/parser/parse_utilcmd.c
***************
*** 1865,1870 **** transformRuleStmt(RuleStmt *stmt, const char *queryString,
--- 1865,1878 ----
  			}
  
  			/*
+ 			 * OLD/NEW is not allowed in CTE queries.
+ 			 */
+ 			if (checkCTEHasOldNew(sub_qry))
+ 				ereport(ERROR,
+ 						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ 						 errmsg("cannot refer to OLD/NEW in CTE query")));
+ 
+ 			/*
  			 * For efficiency's sake, add OLD to the rule action's jointree
  			 * only if it was actually referenced in the statement or qual.
  			 *
*** a/src/backend/rewrite/rewriteManip.c
--- b/src/backend/rewrite/rewriteManip.c
***************
*** 291,296 **** checkExprHasSubLink_walker(Node *node, void *context)
--- 291,320 ----
  	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
  }
  
+ /*
+  * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+  */
+ bool
+ checkCTEHasOldNew(Query *node)
+ {
+ 	ListCell   *l;
+ 
+ 	foreach (l, node->cteList)
+ 	{
+ 		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+ 		int		new_varno = PRS2_NEW_VARNO;
+ 		int		old_varno = PRS2_OLD_VARNO;
+ 
+ 		/* 1 == the top CTE */
+ 		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+ 			return true;
+ 
+ 		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+ 			return true;
+ 	}
+ 
+ 	return false;
+ }
  
  /*
   * OffsetVarNodes - adjust Vars when appending one query's RT to another
*** a/src/backend/utils/adt/ruleutils.c
--- b/src/backend/utils/adt/ruleutils.c
***************
*** 3345,3350 **** get_insert_query_def(Query *query, deparse_context *context)
--- 3345,3353 ----
  	ListCell   *l;
  	List	   *strippedexprs;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
  	 * a single RTE for the SELECT or VALUES.
***************
*** 3482,3487 **** get_update_query_def(Query *query, deparse_context *context)
--- 3485,3493 ----
  	RangeTblEntry *rte;
  	ListCell   *l;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with UPDATE relname SET
  	 */
***************
*** 3563,3568 **** get_delete_query_def(Query *query, deparse_context *context)
--- 3569,3577 ----
  	StringInfo	buf = context->buf;
  	RangeTblEntry *rte;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with DELETE FROM relname
  	 */
*** a/src/include/nodes/parsenodes.h
--- b/src/include/nodes/parsenodes.h
***************
*** 896,901 **** typedef struct InsertStmt
--- 896,902 ----
  	List	   *cols;			/* optional: names of the target columns */
  	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } InsertStmt;
  
  /* ----------------------
***************
*** 909,914 **** typedef struct DeleteStmt
--- 910,916 ----
  	List	   *usingClause;	/* optional using clause for more tables */
  	Node	   *whereClause;	/* qualifications */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } DeleteStmt;
  
  /* ----------------------
***************
*** 923,928 **** typedef struct UpdateStmt
--- 925,931 ----
  	Node	   *whereClause;	/* qualifications */
  	List	   *fromClause;		/* optional from clause for more tables */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } UpdateStmt;
  
  /* ----------------------
*** a/src/include/rewrite/rewriteManip.h
--- b/src/include/rewrite/rewriteManip.h
***************
*** 56,61 **** extern int	locate_windowfunc(Node *node);
--- 56,62 ----
  extern bool checkExprHasAggs(Node *node);
  extern bool checkExprHasWindowFuncs(Node *node);
  extern bool checkExprHasSubLink(Node *node);
+ extern bool checkCTEHasOldNew(Query *node);
  
  extern Node *replace_rte_variables(Node *node,
  					  int target_varno, int sublevels_up,
*** a/src/test/regress/expected/with.out
--- b/src/test/regress/expected/with.out
***************
*** 738,743 **** WITH RECURSIVE
--- 738,820 ----
  (54 rows)
  
  --
+ -- WITH on top of a DML statement
+ --
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+  a  
+ ----
+  21
+  22
+  23
+  24
+  25
+  26
+  27
+  28
+  29
+  30
+ (10 rows)
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ SELECT * FROM y;
+  a  
+ ----
+   1
+   2
+   3
+   4
+   5
+   6
+   7
+   8
+   9
+  10
+ (10 rows)
+ 
+ --
  -- error cases
  --
  -- INTERSECT
***************
*** 774,781 **** WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  ERROR:  recursive reference to query "x" must not appear within its non-recursive term
  LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                                ^
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
  -- LEFT JOIN
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
  	UNION ALL
--- 851,856 ----
***************
*** 912,917 **** ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
--- 987,997 ----
  LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                     ^
  HINT:  Cast the output of the non-recursive term to the correct type.
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ ERROR:  cannot refer to OLD/NEW in CTE query
  --
  -- test for bug #4902
  --
*** a/src/test/regress/sql/with.sql
--- b/src/test/regress/sql/with.sql
***************
*** 339,344 **** WITH RECURSIVE
--- 339,371 ----
   SELECT * FROM z;
  
  --
+ -- WITH on top of a DML statement
+ --
+ 
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ 
+ SELECT * FROM y;
+ 
+ --
  -- error cases
  --
  
***************
*** 364,372 **** WITH RECURSIVE x(n) AS (SELECT n FROM x)
  WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  	SELECT * FROM x;
  
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
- 
  -- LEFT JOIN
  
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
--- 391,396 ----
***************
*** 470,475 **** WITH RECURSIVE foo(i) AS
--- 494,504 ----
     SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
  SELECT * FROM foo;
  
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ 
  --
  -- test for bug #4902
  --
#2Robert Haas
robertmhaas@gmail.com
In reply to: Hitoshi Harada (#1)
Re: top-level DML under CTEs

On Mon, Sep 13, 2010 at 9:15 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

The patch attached is based on the one rejected at the last CF for 9.0
last year.

http://archives.postgresql.org/message-id/16303.1266023203@sss.pgh.pa.us

This patch implements the feature that allows top-level DMLs under CTE
WITH clause. For example:

WITH t AS (SELECT * FROM x)
UPDATE y SET val = t.val FROM t
WHERE y.key = t.key;

This feature is part of writeable CTEs proposed by David Fetter originally.

Thanks for pursuing this. I think this will be a useful feature if we
can get it committed, and plus David Fetter will be very, very happy.
:-)

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

#3Merlin Moncure
mmoncure@gmail.com
In reply to: Robert Haas (#2)
Re: top-level DML under CTEs

On Mon, Sep 13, 2010 at 9:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Mon, Sep 13, 2010 at 9:15 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

The patch attached is based on the one rejected at the last CF for 9.0
last year.

http://archives.postgresql.org/message-id/16303.1266023203@sss.pgh.pa.us

This patch implements the feature that allows top-level DMLs under CTE
WITH clause. For example:

WITH t AS (SELECT * FROM x)
UPDATE y SET val = t.val FROM t
WHERE y.key = t.key;

This feature is part of writeable CTEs proposed by David Fetter originally.

Thanks for pursuing this.  I think this will be a useful feature if we
can get it committed, and plus David Fetter will be very, very happy.
:-)

Just to be clear, the attached patch is missing the part of the wCTE
that allows queries of the form:
WITH foo AS (DELETE * FROM bar RETURNING *) <any query using foo>

IOW, your CTE query has to be a select. This is still highly useful
however. The patch itself looks very clean after a quick glance
(which is all I can offer ATM unfortunately).

merlin

#4Robert Haas
robertmhaas@gmail.com
In reply to: Merlin Moncure (#3)
Re: top-level DML under CTEs

On Mon, Sep 13, 2010 at 2:43 PM, Merlin Moncure <mmoncure@gmail.com> wrote:

Just to be clear, the attached patch is missing the part of the wCTE
that allows queries of the form:
WITH foo AS (DELETE * FROM bar RETURNING *) <any query using foo>

Understood.

IOW, your CTE query has to be a select.  This is still highly useful
however.

Agreed.

The patch itself looks very clean after a quick glance
(which is all I can offer ATM unfortunately).

Yeah, I haven't had a chance to look at it either, just wanted to send props.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

#5Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Merlin Moncure (#3)
Re: top-level DML under CTEs

2010/9/14 Merlin Moncure <mmoncure@gmail.com>:

On Mon, Sep 13, 2010 at 9:20 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Mon, Sep 13, 2010 at 9:15 AM, Hitoshi Harada <umi.tanuki@gmail.com> wrote:

The patch attached is based on the one rejected at the last CF for 9.0
last year.

http://archives.postgresql.org/message-id/16303.1266023203@sss.pgh.pa.us

This patch implements the feature that allows top-level DMLs under CTE
WITH clause. For example:

WITH t AS (SELECT * FROM x)
UPDATE y SET val = t.val FROM t
WHERE y.key = t.key;

This feature is part of writeable CTEs proposed by David Fetter originally.

Thanks for pursuing this.  I think this will be a useful feature if we
can get it committed, and plus David Fetter will be very, very happy.
:-)

Just to be clear, the attached patch is missing the part of the wCTE

Yes, the main part of wCTE that allows DMLs in WITH is still under
Marko Tikkaja. To work parallel, we split the tasks into pieces.

Regards,

--
Hitoshi Harada

#6Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Hitoshi Harada (#1)
Re: top-level DML under CTEs

On 2010-09-13 4:15 PM +0300, Hitoshi Harada wrote:

1. WITH clause atop INSERT
Although the previous discussion got the consensus that we forbid WITH
atop INSERT, it seems to me that it can be allowed. I managed to do it
by treating the top WITH clause (of INSERT) as if the one of SELECT
(or VALUES).

In the email you referred to, Tom was concerned about the case where
these WITH lists have different RECURSIVE declarations. This patch
makes both RECURSIVE if either of them is. I can think of cases where
that might lead to surprising behaviour, but the chances of any of those
happening in real life seem pretty slim.

It is possible to disallow the CTE over INSERT statement,
but the lack for INSERT, though there are for UPDATE and DELETE,
sounds inconsistent enough.

Also because wCTEs are not allowed below the top level, not being able
to use INSERT as the top level statement would force people to wrap that
INSERT in another CTE.

Regards,
Marko Tiikkaja

#7Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Marko Tiikkaja (#6)
Re: top-level DML under CTEs

2010/9/15 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

On 2010-09-13 4:15 PM +0300, Hitoshi Harada wrote:

1. WITH clause atop INSERT
Although the previous discussion got the consensus that we forbid WITH
atop INSERT, it seems to me that it can be allowed. I managed to do it
by treating the top WITH clause (of INSERT) as if the one of SELECT
(or VALUES).

In the email you referred to, Tom was concerned about the case where these
WITH lists have different RECURSIVE declarations.  This patch makes both
RECURSIVE if either of them is.  I can think of cases where that might lead
to surprising behaviour, but the chances of any of those happening in real
life seem pretty slim.

I might not understand the RECURSIVE issue correctly. I put my effort
to make such query

WITH RECURSIVE r AS (SELECT 1 i UNION ALL SELECT i + 1 FROM r WHERE i
< 10) INSERT INTO WITH t AS (SELECT 0) VALUES((SELECT * FROM r LIMIT
1)),((SELECT * FROM t));

look like

INSERT INTO WITH RECURSIVE r AS (SELECT 1 i UNION ALL SELECT i + 1
FROM r WHERE i < 10), t AS (SELECT 0) VALUES((SELECT * FROM r LIMIT
1)),((SELECT * FROM t));

Does that cause surprising behavior?

but the chances of any of those happening in real
life seem pretty slim.

The OLD/NEW issue is also near impossible to be problem in the real
life, except for the misleading error message. But once users see the
non-understandable behavior, they make lines to claim as it's a "bug".
So we need to put effort to avoid it as possible, I believe.

Regards,

--
Hitoshi Harada

#8Tom Lane
tgl@sss.pgh.pa.us
In reply to: Hitoshi Harada (#7)
Re: top-level DML under CTEs

Hitoshi Harada <umi.tanuki@gmail.com> writes:

2010/9/15 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

In the email you referred to, Tom was concerned about the case where these
WITH lists have different RECURSIVE declarations. �This patch makes both
RECURSIVE if either of them is. �I can think of cases where that might lead
to surprising behaviour, but the chances of any of those happening in real
life seem pretty slim.

Does that cause surprising behavior?

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

It would probably be all right to combine the cases internally, at the
rewriter or planner stage. It's not okay to do it in the parser, not
even after doing parse analysis of the individual CTEs, because then it
would be impossible for ruleutils.c to reverse-list the query correctly.

regards, tom lane

#9Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Tom Lane (#8)
Re: top-level DML under CTEs

On 2010-09-14 10:51 PM, Tom Lane wrote:

Hitoshi Harada<umi.tanuki@gmail.com> writes:

2010/9/15 Marko Tiikkaja<marko.tiikkaja@cs.helsinki.fi>:

In the email you referred to, Tom was concerned about the case where these
WITH lists have different RECURSIVE declarations. This patch makes both
RECURSIVE if either of them is. I can think of cases where that might lead
to surprising behaviour, but the chances of any of those happening in real
life seem pretty slim.

Does that cause surprising behavior?

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

The worst I can think of is:

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

t will actually be populated with the results of the CTE, not the table foo.

I don't think this is a huge problem in real life, but if someone thinks
otherwise, I think we could just error out if the lists have a different
RECURSIVE definition.

Anyone have a worse example? Thoughts?

Regards,
Marko Tiikkaja

#10Robert Haas
robertmhaas@gmail.com
In reply to: Marko Tiikkaja (#9)
Re: top-level DML under CTEs

On Tue, Sep 14, 2010 at 4:28 PM, Marko Tiikkaja
<marko.tiikkaja@cs.helsinki.fi> wrote:

On 2010-09-14 10:51 PM, Tom Lane wrote:

Hitoshi Harada<umi.tanuki@gmail.com>  writes:

2010/9/15 Marko Tiikkaja<marko.tiikkaja@cs.helsinki.fi>:

In the email you referred to, Tom was concerned about the case where
these
WITH lists have different RECURSIVE declarations.  This patch makes both
RECURSIVE if either of them is.  I can think of cases where that might
lead
to surprising behaviour, but the chances of any of those happening in
real
life seem pretty slim.

Does that cause surprising behavior?

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

The worst I can think of is:

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

t will actually be populated with the results of the CTE, not the table foo.

Unless I'm confused, that seems pretty clearly wrong.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

#11Tom Lane
tgl@sss.pgh.pa.us
In reply to: Marko Tiikkaja (#9)
Re: top-level DML under CTEs

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:

On 2010-09-14 10:51 PM, Tom Lane wrote:

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

The worst I can think of is:

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

t will actually be populated with the results of the CTE, not the table foo.

I don't think this is a huge problem in real life, but if someone thinks
otherwise, I think we could just error out if the lists have a different
RECURSIVE definition.

Wrong is wrong. Doesn't matter whether it's "a huge problem in real life".

Why is it so difficult to do this correctly?

regards, tom lane

#12Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Marko Tiikkaja (#9)
Re: top-level DML under CTEs

2010/9/15 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

On 2010-09-14 10:51 PM, Tom Lane wrote:

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

The worst I can think of is:

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

t will actually be populated with the results of the CTE, not the table foo.

Hmmm, that's true. But it seems unrelated to RECURSIVE option, right?

Regards,

--
Hitoshi Harada

#13Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Tom Lane (#11)
Re: top-level DML under CTEs

2010/9/15 Tom Lane <tgl@sss.pgh.pa.us>:

Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> writes:

On 2010-09-14 10:51 PM, Tom Lane wrote:

My recollection is that whether a CTE is marked RECURSIVE or not affects
its scope of visibility, so that confusing the two cases can result in
flat-out incorrect parser behavior.

The worst I can think of is:

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

t will actually be populated with the results of the CTE, not the table foo.

I don't think this is a huge problem in real life, but if someone thinks
otherwise, I think we could just error out if the lists have a different
RECURSIVE definition.

Wrong is wrong.  Doesn't matter whether it's "a huge problem in real life".

Why is it so difficult to do this correctly?

Because INSERT INTO ... (SELECT|VALUES) is already a collection of
kludge (as comments say). It was possible to parse the two WITHs
separately, but it results in ambiguous naming issue;
parseWithClause() asserts there's only one WITH clause in the Stmt and
detects duplicated CTE name in it. It seems possible to call
parseWithClause() twice by cheating ParseState and to try to find name
duplication outside it, though it is another kludge :-(

Now that we find the worst situation, I start to think I have to take
the kludy way anyway.

Regards,

--
Hitoshi Harada

#14Tom Lane
tgl@sss.pgh.pa.us
In reply to: Hitoshi Harada (#13)
Re: top-level DML under CTEs

Hitoshi Harada <umi.tanuki@gmail.com> writes:

2010/9/15 Tom Lane <tgl@sss.pgh.pa.us>:

Why is it so difficult to do this correctly?

Because INSERT INTO ... (SELECT|VALUES) is already a collection of
kludge (as comments say). It was possible to parse the two WITHs
separately, but it results in ambiguous naming issue;
parseWithClause() asserts there's only one WITH clause in the Stmt and
detects duplicated CTE name in it.

Well, I would think that the no-duplication rule applies to each WITH
list separately, not both together. If you do something like

with t1 as (select * from foo)
select * from
(with t2 as (select * from foo)
select * from t1, t2) ss;

there's no expectation that the WITH clauses can't both define the same
name.

regards, tom lane

#15Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Tom Lane (#14)
Re: top-level DML under CTEs

2010/9/15 Tom Lane <tgl@sss.pgh.pa.us>:

Hitoshi Harada <umi.tanuki@gmail.com> writes:

2010/9/15 Tom Lane <tgl@sss.pgh.pa.us>:

Why is it so difficult to do this correctly?

Because INSERT INTO ... (SELECT|VALUES) is already a collection of
kludge (as comments say). It was possible to parse the two WITHs
separately, but it results in ambiguous naming issue;
parseWithClause() asserts there's only one WITH clause in the Stmt and
detects duplicated CTE name in it.

Well, I would think that the no-duplication rule applies to each WITH
list separately, not both together.  If you do something like

with t1 as (select * from foo)
 select * from
   (with t2 as (select * from foo)
      select * from t1, t2) ss;

Well, I didn't know it is allowed. That would look like the way to go.

Regards,

--
Hitoshi Harada

#16Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Hitoshi Harada (#15)
1 attachment(s)
Re: top-level DML under CTEs

2010/9/15 Hitoshi Harada <umi.tanuki@gmail.com>:

2010/9/15 Tom Lane <tgl@sss.pgh.pa.us>:

Well, I would think that the no-duplication rule applies to each WITH
list separately, not both together.  If you do something like

with t1 as (select * from foo)
 select * from
   (with t2 as (select * from foo)
      select * from t1, t2) ss;

Well, I didn't know it is allowed. That would look like the way to go.

I made changes to the previous version, so that it avoids to resolve
CTE name duplication.

regression=# with t as (select 1 as i) insert into z with t as(select
2 as i )values ((select * from t));
INSERT 0 1
Time: 1.656 ms
regression=# table z;
f3
----
2
(1 row)

Also, the sample Marko gave is OK.

CREATE TABLE foo(a int);

WITH t AS (SELECT * FROM foo)
INSERT INTO bar
WITH RECURSIVE foo (SELECT 1 AS a)
SELECT * FROM t;

Hope this covers all the cases.

Regards,

--
Hitoshi Harada

Attachments:

toplevel-dml-cte.20100917.patchapplication/octet-stream; name=toplevel-dml-cte.20100917.patchDownload
*** a/doc/src/sgml/ref/delete.sgml
--- b/doc/src/sgml/ref/delete.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
      [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
***************
*** 84,89 **** DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
--- 85,102 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><literal>ONLY</></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/insert.sgml
--- b/doc/src/sgml/ref/insert.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
      { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
      [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
***************
*** 85,90 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
--- 86,109 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+      <para>
+       It is possible that <literal>SELECT</literal> query also has
+       <literal>WITH</literal>.  In this case the two
+       <replaceable>with_query</replaceable> can be referred from
+       the <literal>SELECT</literal> query.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/update.sgml
--- b/doc/src/sgml/ref/update.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
            ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
***************
*** 80,85 **** UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
--- 81,98 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***************
*** 2287,2292 **** _copyInsertStmt(InsertStmt *from)
--- 2287,2293 ----
  	COPY_NODE_FIELD(cols);
  	COPY_NODE_FIELD(selectStmt);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2300,2305 **** _copyDeleteStmt(DeleteStmt *from)
--- 2301,2307 ----
  	COPY_NODE_FIELD(usingClause);
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2314,2319 **** _copyUpdateStmt(UpdateStmt *from)
--- 2316,2322 ----
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(fromClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
*** a/src/backend/nodes/equalfuncs.c
--- b/src/backend/nodes/equalfuncs.c
***************
*** 889,894 **** _equalInsertStmt(InsertStmt *a, InsertStmt *b)
--- 889,895 ----
  	COMPARE_NODE_FIELD(cols);
  	COMPARE_NODE_FIELD(selectStmt);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 900,905 **** _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
--- 901,907 ----
  	COMPARE_NODE_FIELD(usingClause);
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 912,917 **** _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
--- 914,920 ----
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(fromClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
*** a/src/backend/parser/analyze.c
--- b/src/backend/parser/analyze.c
***************
*** 282,287 **** transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
--- 282,294 ----
  
  	qry->commandType = CMD_DELETE;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/* set up range table with just the result rel */
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
***************
*** 342,347 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
--- 349,361 ----
  	qry->commandType = CMD_INSERT;
  	pstate->p_is_insert = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/*
  	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
  	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
***************
*** 366,373 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		pstate->p_relnamespace = NIL;
  		sub_varnamespace = pstate->p_varnamespace;
  		pstate->p_varnamespace = NIL;
- 		/* There can't be any outer WITH to worry about */
- 		Assert(pstate->p_ctenamespace == NIL);
  	}
  	else
  	{
--- 380,385 ----
***************
*** 511,518 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		/* process the WITH clause */
  		if (selectStmt->withClause)
  		{
! 			qry->hasRecursive = selectStmt->withClause->recursive;
  			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
  		}
  
  		foreach(lc, selectStmt->valuesLists)
--- 523,542 ----
  		/* process the WITH clause */
  		if (selectStmt->withClause)
  		{
! 			List   *cteList = qry->cteList;
! 
! 			/* pretend CTE transform is brand-new */
! 			pstate->p_ctenamespace = NIL;
! 			pstate->p_future_ctes = NIL;
! 			qry->hasRecursive |= selectStmt->withClause->recursive;
  			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
+ 			/*
+ 			 * To match the behavior to generalSelect case,
+ 			 * SELECT's CTE shoud come first. It is also to
+ 			 * save both(top-level and SELECT-level CTEs
+ 			 * in the pstate->p_ctenamespace.
+ 			 */
+ 			qry->cteList = list_concat(qry->cteList, cteList);
  		}
  
  		foreach(lc, selectStmt->valuesLists)
***************
*** 611,618 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		/* process the WITH clause */
  		if (selectStmt->withClause)
  		{
! 			qry->hasRecursive = selectStmt->withClause->recursive;
  			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
  		}
  
  		/* Do basic expression transformation (same as a ROW() expr) */
--- 635,654 ----
  		/* process the WITH clause */
  		if (selectStmt->withClause)
  		{
! 			List   *cteList = qry->cteList;
! 
! 			/* pretend CTE transform is brand-new */
! 			pstate->p_ctenamespace = NIL;
! 			pstate->p_future_ctes = NIL;
! 			qry->hasRecursive |= selectStmt->withClause->recursive;
  			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
+ 			/*
+ 			 * To match the behavior to generalSelect case,
+ 			 * SELECT's CTE shoud come first. It is also to
+ 			 * save both(top-level and SELECT-level CTEs
+ 			 * in the pstate->p_ctenamespace.
+ 			 */
+ 			qry->cteList = list_concat(qry->cteList, cteList);
  		}
  
  		/* Do basic expression transformation (same as a ROW() expr) */
***************
*** 1726,1731 **** transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
--- 1762,1774 ----
  	qry->commandType = CMD_UPDATE;
  	pstate->p_is_update = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
  										 true,
*** a/src/backend/parser/gram.y
--- b/src/backend/parser/gram.y
***************
*** 430,436 **** static TypeName *TableFuncTypeName(List *columns);
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
--- 430,436 ----
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause opt_with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
***************
*** 7129,7139 **** DeallocateStmt: DEALLOCATE name
   *****************************************************************************/
  
  InsertStmt:
! 			INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$4->relation = $3;
! 					$4->returningList = $5;
! 					$$ = (Node *) $4;
  				}
  		;
  
--- 7129,7140 ----
   *****************************************************************************/
  
  InsertStmt:
! 			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$5->relation = $4;
! 					$5->returningList = $6;
! 					$5->withClause = $1;
! 					$$ = (Node *) $5;
  				}
  		;
  
***************
*** 7189,7202 **** returning_clause:
   *
   *****************************************************************************/
  
! DeleteStmt: DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $3;
! 					n->usingClause = $4;
! 					n->whereClause = $5;
! 					n->returningList = $6;
  					$$ = (Node *)n;
  				}
  		;
--- 7190,7204 ----
   *
   *****************************************************************************/
  
! DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $4;
! 					n->usingClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7251,7268 **** opt_nowait:	NOWAIT							{ $$ = TRUE; }
   *
   *****************************************************************************/
  
! UpdateStmt: UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $2;
! 					n->targetList = $4;
! 					n->fromClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
  					$$ = (Node *)n;
  				}
  		;
--- 7253,7271 ----
   *
   *****************************************************************************/
  
! UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $3;
! 					n->targetList = $5;
! 					n->fromClause = $6;
! 					n->whereClause = $7;
! 					n->returningList = $8;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7604,7609 **** common_table_expr:  name opt_name_list AS select_with_parens
--- 7607,7618 ----
  			}
  		;
  
+ opt_with_clause:
+ 		with_clause								{ $$ = $1; }
+ 		| /*EMPTY*/								{ $$ = NULL; }
+ 		;
+ 
+ 
  into_clause:
  			INTO OptTempTableName
  				{
*** a/src/backend/parser/parse_utilcmd.c
--- b/src/backend/parser/parse_utilcmd.c
***************
*** 1865,1870 **** transformRuleStmt(RuleStmt *stmt, const char *queryString,
--- 1865,1878 ----
  			}
  
  			/*
+ 			 * OLD/NEW is not allowed in CTE queries.
+ 			 */
+ 			if (checkCTEHasOldNew(sub_qry))
+ 				ereport(ERROR,
+ 						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ 						 errmsg("cannot refer to OLD/NEW in CTE query")));
+ 
+ 			/*
  			 * For efficiency's sake, add OLD to the rule action's jointree
  			 * only if it was actually referenced in the statement or qual.
  			 *
*** a/src/backend/rewrite/rewriteManip.c
--- b/src/backend/rewrite/rewriteManip.c
***************
*** 291,296 **** checkExprHasSubLink_walker(Node *node, void *context)
--- 291,320 ----
  	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
  }
  
+ /*
+  * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+  */
+ bool
+ checkCTEHasOldNew(Query *node)
+ {
+ 	ListCell   *l;
+ 
+ 	foreach (l, node->cteList)
+ 	{
+ 		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+ 		int		new_varno = PRS2_NEW_VARNO;
+ 		int		old_varno = PRS2_OLD_VARNO;
+ 
+ 		/* 1 == the top CTE */
+ 		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+ 			return true;
+ 
+ 		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+ 			return true;
+ 	}
+ 
+ 	return false;
+ }
  
  /*
   * OffsetVarNodes - adjust Vars when appending one query's RT to another
*** a/src/backend/utils/adt/ruleutils.c
--- b/src/backend/utils/adt/ruleutils.c
***************
*** 3345,3350 **** get_insert_query_def(Query *query, deparse_context *context)
--- 3345,3353 ----
  	ListCell   *l;
  	List	   *strippedexprs;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
  	 * a single RTE for the SELECT or VALUES.
***************
*** 3482,3487 **** get_update_query_def(Query *query, deparse_context *context)
--- 3485,3493 ----
  	RangeTblEntry *rte;
  	ListCell   *l;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with UPDATE relname SET
  	 */
***************
*** 3563,3568 **** get_delete_query_def(Query *query, deparse_context *context)
--- 3569,3577 ----
  	StringInfo	buf = context->buf;
  	RangeTblEntry *rte;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with DELETE FROM relname
  	 */
*** a/src/include/nodes/parsenodes.h
--- b/src/include/nodes/parsenodes.h
***************
*** 896,901 **** typedef struct InsertStmt
--- 896,902 ----
  	List	   *cols;			/* optional: names of the target columns */
  	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } InsertStmt;
  
  /* ----------------------
***************
*** 909,914 **** typedef struct DeleteStmt
--- 910,916 ----
  	List	   *usingClause;	/* optional using clause for more tables */
  	Node	   *whereClause;	/* qualifications */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } DeleteStmt;
  
  /* ----------------------
***************
*** 923,928 **** typedef struct UpdateStmt
--- 925,931 ----
  	Node	   *whereClause;	/* qualifications */
  	List	   *fromClause;		/* optional from clause for more tables */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } UpdateStmt;
  
  /* ----------------------
*** a/src/include/rewrite/rewriteManip.h
--- b/src/include/rewrite/rewriteManip.h
***************
*** 56,61 **** extern int	locate_windowfunc(Node *node);
--- 56,62 ----
  extern bool checkExprHasAggs(Node *node);
  extern bool checkExprHasWindowFuncs(Node *node);
  extern bool checkExprHasSubLink(Node *node);
+ extern bool checkCTEHasOldNew(Query *node);
  
  extern Node *replace_rte_variables(Node *node,
  					  int target_varno, int sublevels_up,
*** a/src/test/regress/expected/with.out
--- b/src/test/regress/expected/with.out
***************
*** 738,743 **** WITH RECURSIVE
--- 738,820 ----
  (54 rows)
  
  --
+ -- WITH on top of a DML statement
+ --
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+  a  
+ ----
+  21
+  22
+  23
+  24
+  25
+  26
+  27
+  28
+  29
+  30
+ (10 rows)
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ SELECT * FROM y;
+  a  
+ ----
+   1
+   2
+   3
+   4
+   5
+   6
+   7
+   8
+   9
+  10
+ (10 rows)
+ 
+ --
  -- error cases
  --
  -- INTERSECT
***************
*** 774,781 **** WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  ERROR:  recursive reference to query "x" must not appear within its non-recursive term
  LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                                ^
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
  -- LEFT JOIN
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
  	UNION ALL
--- 851,856 ----
***************
*** 912,917 **** ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
--- 987,997 ----
  LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                     ^
  HINT:  Cast the output of the non-recursive term to the correct type.
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ ERROR:  cannot refer to OLD/NEW in CTE query
  --
  -- test for bug #4902
  --
*** a/src/test/regress/sql/with.sql
--- b/src/test/regress/sql/with.sql
***************
*** 339,344 **** WITH RECURSIVE
--- 339,371 ----
   SELECT * FROM z;
  
  --
+ -- WITH on top of a DML statement
+ --
+ 
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ 
+ SELECT * FROM y;
+ 
+ --
  -- error cases
  --
  
***************
*** 364,372 **** WITH RECURSIVE x(n) AS (SELECT n FROM x)
  WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  	SELECT * FROM x;
  
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
- 
  -- LEFT JOIN
  
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
--- 391,396 ----
***************
*** 470,475 **** WITH RECURSIVE foo(i) AS
--- 494,504 ----
     SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
  SELECT * FROM foo;
  
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ 
  --
  -- test for bug #4902
  --
#17Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Hitoshi Harada (#16)
Re: top-level DML under CTEs

On 2010-09-17 4:48 AM, Hitoshi Harada wrote:

2010/9/15 Hitoshi Harada<umi.tanuki@gmail.com>:

Well, I didn't know it is allowed. That would look like the way to go.

I made changes to the previous version, so that it avoids to resolve
CTE name duplication.

This patch still doesn't address the issue Tom raised here:
http://archives.postgresql.org/pgsql-hackers/2010-09/msg00753.php

For WITH .. INSERT .. WITH .. SELECT ..; this patch works OK, but not so
much for VALUES:

=# CREATE RULE barrule AS ON UPDATE TO bar DO INSTEAD
-# WITH RECURSIVE t AS (SELECT -1)
-# INSERT INTO bar
-# WITH t AS (SELECT 1)
-# VALUES((SELECT * FROM t));
CREATE RULE

=# \d bar
Table "public.bar"
Column | Type | Modifiers
--------+---------+-----------
a | integer |
Rules:
barrule AS
ON UPDATE TO bar DO INSTEAD WITH RECURSIVE t AS (
SELECT 1
), t AS (
SELECT (-1)
)
INSERT INTO bar (a) WITH RECURSIVE t AS (
SELECT 1
), t AS (
SELECT (-1)
)

VALUES (( SELECT t."?column?"
FROM t))

Regards,
Marko Tiikkaja

#18Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Marko Tiikkaja (#17)
Re: top-level DML under CTEs

2010/9/23 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

On 2010-09-17 4:48 AM, Hitoshi Harada wrote:

2010/9/15 Hitoshi Harada<umi.tanuki@gmail.com>:

Well, I didn't know it is allowed. That would look like the way to go.

I made changes to the previous version, so that it avoids to resolve
CTE name duplication.

This patch still doesn't address the issue Tom raised here:
http://archives.postgresql.org/pgsql-hackers/2010-09/msg00753.php

For WITH .. INSERT .. WITH .. SELECT ..; this patch works OK, but not so
much for VALUES:

=# CREATE RULE barrule AS ON UPDATE TO bar DO INSTEAD
-# WITH RECURSIVE t AS (SELECT -1)
-# INSERT INTO bar
-# WITH t AS (SELECT 1)
-# VALUES((SELECT * FROM t));
CREATE RULE

=# \d bar
     Table "public.bar"
 Column |  Type   | Modifiers
--------+---------+-----------
 a      | integer |
Rules:
   barrule AS
   ON UPDATE TO bar DO INSTEAD  WITH RECURSIVE t AS (
        SELECT 1
       ), t AS (
        SELECT (-1)
       )
 INSERT INTO bar (a)  WITH RECURSIVE t AS (
                SELECT 1
               ), t AS (
                SELECT (-1)
               )

 VALUES (( SELECT t."?column?"
          FROM t))

I ran the sql and recognized what is wrong. In VALUES case, the WITH
table "t" is copied in one list and shown up in the both of
INSERT-level WITH and SELECT-level WITH. Since the transformation of
WITH clause to cheat postgres is in the parser stage currently, I
wonder if this should be done in the rewriter or the planner stage.

Thanks for the report. Next time, please point the clear problem in
English aside the sample.

Regards,

--
Hitoshi Harada

#19Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Hitoshi Harada (#18)
Re: top-level DML under CTEs

On 2010-09-23 9:12 AM +0300, Hitoshi Harada wrote:

Thanks for the report. Next time, please point the clear problem in
English aside the sample.

I apologize. The problem was exactly the one pointed out in the email I
referred to, so I assumed that further explanation was not necessary.

I will try to be more clear in the future.

Regards,
Marko Tiikkaja

#20Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Hitoshi Harada (#18)
Re: [HACKERS] top-level DML under CTEs

On 2010-09-23 9:12 AM +0300, Hitoshi Harada wrote:

Since the transformation of
WITH clause to cheat postgres is in the parser stage currently, I
wonder if this should be done in the rewriter or the planner stage.

It's been about a week now. Should we expect a new patch soon?

Regards,
Marko Tiikkaja

#21Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Marko Tiikkaja (#20)
Re: [HACKERS] top-level DML under CTEs

2010/9/30 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

On 2010-09-23 9:12 AM +0300, Hitoshi Harada wrote:

Since the transformation of
WITH clause to cheat postgres is in the parser stage currently, I
wonder if this should be done in the rewriter or the planner stage.

It's been about a week now.  Should we expect a new patch soon?

Yep, I'm working it now. You'll see the conclusion in a day or so.

Regards,

--
Hitoshi Harada

#22Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Hitoshi Harada (#21)
1 attachment(s)
Re: [RRR] top-level DML under CTEs

2010/10/1 Hitoshi Harada <umi.tanuki@gmail.com>:

2010/9/30 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

On 2010-09-23 9:12 AM +0300, Hitoshi Harada wrote:

Since the transformation of
WITH clause to cheat postgres is in the parser stage currently, I
wonder if this should be done in the rewriter or the planner stage.

It's been about a week now.  Should we expect a new patch soon?

Yep, I'm working it now. You'll see the conclusion in a day or so.

...and attached is the latest patch. It contains LIMIT etc. bug of
INSERT fixes and I confirmed the barrule case correctly in this
version.

=# CREATE RULE barrule AS ON UPDATE TO bar DO INSTEAD
-# WITH RECURSIVE t AS (SELECT -1)
-# INSERT INTO bar
-# WITH t AS (SELECT 1)
-# VALUES((SELECT * FROM t));

Regards,

--
Hitoshi Harada

Attachments:

toplevel-dml-cte.20101003.diffapplication/octet-stream; name=toplevel-dml-cte.20101003.diffDownload
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
index c87f35c..45a1643 100644
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
     [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
     [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
@@ -84,6 +85,18 @@ DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><literal>ONLY</></term>
     <listitem>
      <para>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
index 6d17ef0..95ca5f8 100644
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
     { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
     [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
@@ -85,6 +86,24 @@ INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+     <para>
+      It is possible that <literal>SELECT</literal> query also has
+      <literal>WITH</literal>.  In this case the two
+      <replaceable>with_query</replaceable> can be referred from
+      the <literal>SELECT</literal> query.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><replaceable class="PARAMETER">table</replaceable></term>
     <listitem>
      <para>
@@ -129,7 +148,8 @@ INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
     <listitem>
      <para>
       The corresponding <replaceable>column</replaceable> will be filled with
-      its default value.
+      its default value. This clause is allowed in a simple VALUES list
+      without additional (LIMIT, etc.) clauses.
      </para>
     </listitem>
    </varlistentry>
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
index c897634..08fb9a0 100644
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
     SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
           ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
@@ -80,6 +81,18 @@ UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><replaceable class="PARAMETER">table</replaceable></term>
     <listitem>
      <para>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 5bd0ef0..18b0b90 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2288,6 +2288,7 @@ _copyInsertStmt(InsertStmt *from)
 	COPY_NODE_FIELD(cols);
 	COPY_NODE_FIELD(selectStmt);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
@@ -2301,6 +2302,7 @@ _copyDeleteStmt(DeleteStmt *from)
 	COPY_NODE_FIELD(usingClause);
 	COPY_NODE_FIELD(whereClause);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
@@ -2315,6 +2317,7 @@ _copyUpdateStmt(UpdateStmt *from)
 	COPY_NODE_FIELD(whereClause);
 	COPY_NODE_FIELD(fromClause);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index c7dd42d..0ea3a31 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -890,6 +890,7 @@ _equalInsertStmt(InsertStmt *a, InsertStmt *b)
 	COMPARE_NODE_FIELD(cols);
 	COMPARE_NODE_FIELD(selectStmt);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
@@ -901,6 +902,7 @@ _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
 	COMPARE_NODE_FIELD(usingClause);
 	COMPARE_NODE_FIELD(whereClause);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
@@ -913,6 +915,7 @@ _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
 	COMPARE_NODE_FIELD(whereClause);
 	COMPARE_NODE_FIELD(fromClause);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 0a93ec7..e2b86c5 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -283,6 +283,13 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
 
 	qry->commandType = CMD_DELETE;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	/* set up range table with just the result rel */
 	qry->resultRelation = setTargetTable(pstate, stmt->relation,
 								  interpretInhOption(stmt->relation->inhOpt),
@@ -343,12 +350,26 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 	qry->commandType = CMD_INSERT;
 	pstate->p_is_insert = true;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	/*
 	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
-	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
-	 * efficiency and so we can handle DEFAULT specifications.
+	 * simple VALUES list, or general SELECT input including complex VALUES.
+	 * We special-case VALUES, both for efficiency and so we can handle
+	 * DEFAULT specifications. In a complex VALUES case, which means the list
+	 * has any of ORDER BY, OFFSET, LIMIT or WITH, we don't accept DEFAULT
+	 * in it; The spec may require it but for now we reject it from point of
+	 * code base and expected use cases.
 	 */
-	isGeneralSelect = (selectStmt && selectStmt->valuesLists == NIL);
+	isGeneralSelect = (selectStmt &&
+		(selectStmt->valuesLists == NIL ||
+		 selectStmt->sortClause || selectStmt->limitOffset ||
+		 selectStmt->limitCount || selectStmt->withClause));
 
 	/*
 	 * If a non-nil rangetable/namespace was passed in, and we are doing
@@ -367,8 +388,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 		pstate->p_relnamespace = NIL;
 		sub_varnamespace = pstate->p_varnamespace;
 		pstate->p_varnamespace = NIL;
-		/* There can't be any outer WITH to worry about */
-		Assert(pstate->p_ctenamespace == NIL);
 	}
 	else
 	{
@@ -509,13 +528,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 		List	   *exprsLists = NIL;
 		int			sublist_length = -1;
 
-		/* process the WITH clause */
-		if (selectStmt->withClause)
-		{
-			qry->hasRecursive = selectStmt->withClause->recursive;
-			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
-		}
-
 		foreach(lc, selectStmt->valuesLists)
 		{
 			List	   *sublist = (List *) lfirst(lc);
@@ -609,13 +621,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 
 		Assert(list_length(valuesLists) == 1);
 
-		/* process the WITH clause */
-		if (selectStmt->withClause)
-		{
-			qry->hasRecursive = selectStmt->withClause->recursive;
-			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
-		}
-
 		/* Do basic expression transformation (same as a ROW() expr) */
 		exprList = transformExpressionList(pstate,
 										   (List *) linitial(valuesLists));
@@ -1036,7 +1041,8 @@ transformValuesClause(ParseState *pstate, SelectStmt *stmt)
 			if (IsA(col, SetToDefault))
 				ereport(ERROR,
 						(errcode(ERRCODE_SYNTAX_ERROR),
-						 errmsg("DEFAULT can only appear in a VALUES list within INSERT"),
+						 errmsg("DEFAULT can only appear in a simple VALUES list within INSERT"),
+						 errhint("A simple VALUES list means it isn't modified by any of ORDER BY, OFFSET, LIMIT or WITH"),
 						 parser_errposition(pstate, exprLocation(col))));
 			colexprs[i] = lappend(colexprs[i], col);
 			i++;
@@ -1785,6 +1791,13 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
 	qry->commandType = CMD_UPDATE;
 	pstate->p_is_update = true;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	qry->resultRelation = setTargetTable(pstate, stmt->relation,
 								  interpretInhOption(stmt->relation->inhOpt),
 										 true,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4054cb1..401c001 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -433,7 +433,7 @@ static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_
 %type <boolean> xml_whitespace_option
 
 %type <node> 	common_table_expr
-%type <with> 	with_clause
+%type <with> 	with_clause opt_with_clause
 %type <list>	cte_list
 
 %type <list>	window_clause window_definition_list opt_partition_clause
@@ -7268,11 +7268,12 @@ DeallocateStmt: DEALLOCATE name
  *****************************************************************************/
 
 InsertStmt:
-			INSERT INTO qualified_name insert_rest returning_clause
+			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
 				{
-					$4->relation = $3;
-					$4->returningList = $5;
-					$$ = (Node *) $4;
+					$5->relation = $4;
+					$5->returningList = $6;
+					$5->withClause = $1;
+					$$ = (Node *) $5;
 				}
 		;
 
@@ -7328,14 +7329,15 @@ returning_clause:
  *
  *****************************************************************************/
 
-DeleteStmt: DELETE_P FROM relation_expr_opt_alias
+DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
 			using_clause where_or_current_clause returning_clause
 				{
 					DeleteStmt *n = makeNode(DeleteStmt);
-					n->relation = $3;
-					n->usingClause = $4;
-					n->whereClause = $5;
-					n->returningList = $6;
+					n->relation = $4;
+					n->usingClause = $5;
+					n->whereClause = $6;
+					n->returningList = $7;
+					n->withClause = $1;
 					$$ = (Node *)n;
 				}
 		;
@@ -7390,18 +7392,19 @@ opt_nowait:	NOWAIT							{ $$ = TRUE; }
  *
  *****************************************************************************/
 
-UpdateStmt: UPDATE relation_expr_opt_alias
+UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
 			SET set_clause_list
 			from_clause
 			where_or_current_clause
 			returning_clause
 				{
 					UpdateStmt *n = makeNode(UpdateStmt);
-					n->relation = $2;
-					n->targetList = $4;
-					n->fromClause = $5;
-					n->whereClause = $6;
-					n->returningList = $7;
+					n->relation = $3;
+					n->targetList = $5;
+					n->fromClause = $6;
+					n->whereClause = $7;
+					n->returningList = $8;
+					n->withClause = $1;
 					$$ = (Node *)n;
 				}
 		;
@@ -7743,6 +7746,12 @@ common_table_expr:  name opt_name_list AS select_with_parens
 			}
 		;
 
+opt_with_clause:
+		with_clause								{ $$ = $1; }
+		| /*EMPTY*/								{ $$ = NULL; }
+		;
+
+
 into_clause:
 			INTO OptTempTableName
 				{
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 37ca331..663754a 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1868,6 +1868,14 @@ transformRuleStmt(RuleStmt *stmt, const char *queryString,
 			}
 
 			/*
+			 * OLD/NEW is not allowed in CTE queries.
+			 */
+			if (checkCTEHasOldNew(sub_qry))
+				ereport(ERROR,
+						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+						 errmsg("cannot refer to OLD/NEW in CTE query")));
+
+			/*
 			 * For efficiency's sake, add OLD to the rule action's jointree
 			 * only if it was actually referenced in the statement or qual.
 			 *
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index 5db2522..ca6d494 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -291,6 +291,30 @@ checkExprHasSubLink_walker(Node *node, void *context)
 	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
 }
 
+/*
+ * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+ */
+bool
+checkCTEHasOldNew(Query *node)
+{
+	ListCell   *l;
+
+	foreach (l, node->cteList)
+	{
+		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+		int		new_varno = PRS2_NEW_VARNO;
+		int		old_varno = PRS2_OLD_VARNO;
+
+		/* 1 == the top CTE */
+		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+			return true;
+
+		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+			return true;
+	}
+
+	return false;
+}
 
 /*
  * OffsetVarNodes - adjust Vars when appending one query's RT to another
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 578b9ce..cec5e19 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -3345,6 +3345,9 @@ get_insert_query_def(Query *query, deparse_context *context)
 	ListCell   *l;
 	List	   *strippedexprs;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
 	 * a single RTE for the SELECT or VALUES.
@@ -3482,6 +3485,9 @@ get_update_query_def(Query *query, deparse_context *context)
 	RangeTblEntry *rte;
 	ListCell   *l;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * Start the query with UPDATE relname SET
 	 */
@@ -3563,6 +3569,9 @@ get_delete_query_def(Query *query, deparse_context *context)
 	StringInfo	buf = context->buf;
 	RangeTblEntry *rte;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * Start the query with DELETE FROM relname
 	 */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index b2f0fef..d93d759 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -896,6 +896,7 @@ typedef struct InsertStmt
 	List	   *cols;			/* optional: names of the target columns */
 	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } InsertStmt;
 
 /* ----------------------
@@ -909,6 +910,7 @@ typedef struct DeleteStmt
 	List	   *usingClause;	/* optional using clause for more tables */
 	Node	   *whereClause;	/* qualifications */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } DeleteStmt;
 
 /* ----------------------
@@ -923,6 +925,7 @@ typedef struct UpdateStmt
 	Node	   *whereClause;	/* qualifications */
 	List	   *fromClause;		/* optional from clause for more tables */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } UpdateStmt;
 
 /* ----------------------
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
index 8daea6e..7a1366b 100644
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -56,6 +56,7 @@ extern int	locate_windowfunc(Node *node);
 extern bool checkExprHasAggs(Node *node);
 extern bool checkExprHasWindowFuncs(Node *node);
 extern bool checkExprHasSubLink(Node *node);
+extern bool checkCTEHasOldNew(Query *node);
 
 extern Node *replace_rte_variables(Node *node,
 					  int target_varno, int sublevels_up,
diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out
index e46ed78..a50148b 100644
--- a/src/test/regress/expected/with.out
+++ b/src/test/regress/expected/with.out
@@ -738,6 +738,83 @@ WITH RECURSIVE
 (54 rows)
 
 --
+-- WITH on top of a DML statement
+--
+CREATE TEMPORARY TABLE y (a INTEGER);
+INSERT INTO y SELECT generate_series(1, 10);
+WITH t AS (
+	SELECT a FROM y
+)
+INSERT INTO y
+SELECT a+20 FROM t RETURNING *;
+ a  
+----
+ 21
+ 22
+ 23
+ 24
+ 25
+ 26
+ 27
+ 28
+ 29
+ 30
+(10 rows)
+
+WITH t AS (
+	SELECT a FROM y
+)
+UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ a  
+----
+ 11
+ 12
+ 13
+ 14
+ 15
+ 16
+ 17
+ 18
+ 19
+ 20
+(10 rows)
+
+WITH RECURSIVE t(a) AS (
+	SELECT 11
+	UNION ALL
+	SELECT a+1 FROM t WHERE a < 50
+)
+DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ a  
+----
+ 11
+ 12
+ 13
+ 14
+ 15
+ 16
+ 17
+ 18
+ 19
+ 20
+(10 rows)
+
+SELECT * FROM y;
+ a  
+----
+  1
+  2
+  3
+  4
+  5
+  6
+  7
+  8
+  9
+ 10
+(10 rows)
+
+--
 -- error cases
 --
 -- INTERSECT
@@ -774,8 +851,6 @@ WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
 ERROR:  recursive reference to query "x" must not appear within its non-recursive term
 LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                               ^
-CREATE TEMPORARY TABLE y (a INTEGER);
-INSERT INTO y SELECT generate_series(1, 10);
 -- LEFT JOIN
 WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
 	UNION ALL
@@ -912,6 +987,11 @@ ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
 LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                    ^
 HINT:  Cast the output of the non-recursive term to the correct type.
+-- disallow OLD/NEW reference in CTE
+CREATE TEMPORARY TABLE x (n integer);
+CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+    WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ERROR:  cannot refer to OLD/NEW in CTE query
 --
 -- test for bug #4902
 --
diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql
index 2cbaa42..7fc70c0 100644
--- a/src/test/regress/sql/with.sql
+++ b/src/test/regress/sql/with.sql
@@ -339,6 +339,33 @@ WITH RECURSIVE
  SELECT * FROM z;
 
 --
+-- WITH on top of a DML statement
+--
+
+CREATE TEMPORARY TABLE y (a INTEGER);
+INSERT INTO y SELECT generate_series(1, 10);
+
+WITH t AS (
+	SELECT a FROM y
+)
+INSERT INTO y
+SELECT a+20 FROM t RETURNING *;
+
+WITH t AS (
+	SELECT a FROM y
+)
+UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+
+WITH RECURSIVE t(a) AS (
+	SELECT 11
+	UNION ALL
+	SELECT a+1 FROM t WHERE a < 50
+)
+DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+
+SELECT * FROM y;
+
+--
 -- error cases
 --
 
@@ -364,9 +391,6 @@ WITH RECURSIVE x(n) AS (SELECT n FROM x)
 WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
 	SELECT * FROM x;
 
-CREATE TEMPORARY TABLE y (a INTEGER);
-INSERT INTO y SELECT generate_series(1, 10);
-
 -- LEFT JOIN
 
 WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
@@ -470,6 +494,11 @@ WITH RECURSIVE foo(i) AS
    SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
 SELECT * FROM foo;
 
+-- disallow OLD/NEW reference in CTE
+CREATE TEMPORARY TABLE x (n integer);
+CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+    WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+
 --
 -- test for bug #4902
 --
#23Erik Rijkers
er@xs4all.nl
In reply to: Hitoshi Harada (#22)
Re: [HACKERS] top-level DML under CTEs

On Sun, October 3, 2010 15:47, Hitoshi Harada wrote:

[...]

...and attached is the latest patch. It contains LIMIT etc. bug of
INSERT fixes and I confirmed the barrule case correctly in this
version.

(HEAD from git://git.postgresql.org/git/postgresql.git)

The patch applies only with error.
If that error is ignored, the regression 'with' test failes.
If that is also ignored, it runs.

I thought I'd give you the errors anyway:

patch --strip=1 < toplevel-dml-cte.20101003.diff
patching file doc/src/sgml/ref/delete.sgml
patching file doc/src/sgml/ref/insert.sgml
patching file doc/src/sgml/ref/update.sgml
patching file src/backend/nodes/copyfuncs.c
patching file src/backend/nodes/equalfuncs.c
patching file src/backend/parser/analyze.c
Hunk #2 FAILED at 350.
Hunk #3 succeeded at 397 (offset 9 lines).
Hunk #5 succeeded at 630 (offset 9 lines).
Hunk #7 succeeded at 1800 (offset 9 lines).
1 out of 7 hunks FAILED -- saving rejects to file src/backend/parser/analyze.c.rej
patching file src/backend/parser/gram.y
patching file src/backend/parser/parse_utilcmd.c
patching file src/backend/rewrite/rewriteManip.c
patching file src/backend/utils/adt/ruleutils.c
patching file src/include/nodes/parsenodes.h
patching file src/include/rewrite/rewriteManip.h
patching file src/test/regress/expected/with.out
patching file src/test/regress/sql/with.sql
-------------------8<-------------------
***************
*** 343,354 ****
qry->commandType = CMD_INSERT;
pstate->p_is_insert = true;

/*
* We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
- * VALUES list, or general SELECT input. We special-case VALUES, both for
- * efficiency and so we can handle DEFAULT specifications.
*/
- isGeneralSelect = (selectStmt && selectStmt->valuesLists == NIL);

  	/*
  	 * If a non-nil rangetable/namespace was passed in, and we are doing
--- 350,375 ----
  	qry->commandType = CMD_INSERT;
  	pstate->p_is_insert = true;
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+
  	/*
  	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
+ 	 * simple VALUES list, or general SELECT input including complex VALUES.
+ 	 * We special-case VALUES, both for efficiency and so we can handle
+ 	 * DEFAULT specifications. In a complex VALUES case, which means the list
+ 	 * has any of ORDER BY, OFFSET, LIMIT or WITH, we don't accept DEFAULT
+ 	 * in it; The spec may require it but for now we reject it from point of
+ 	 * code base and expected use cases.
  	 */
+ 	isGeneralSelect = (selectStmt &&
+ 		(selectStmt->valuesLists == NIL ||
+ 		 selectStmt->sortClause || selectStmt->limitOffset ||
+ 		 selectStmt->limitCount || selectStmt->withClause));

/*
* If a non-nil rangetable/namespace was passed in, and we are doing
-------------------8<-------------------

Continuing after that error:
make OK;
make check:

[...]
largeobject ... ok
with ... FAILED
xml ... ok
[...]

regression.diffs:

*** /var/data1/pg_stuff/pg_sandbox/pgsql.dml_cte/src/test/regress/expected/with.out	2010-10-04
13:25:26.000000000 +0200
--- /var/data1/pg_stuff/pg_sandbox/pgsql.dml_cte/src/test/regress/results/with.out	2010-10-04
13:28:20.000000000 +0200
***************
*** 747,783 ****
  )
  INSERT INTO y
  SELECT a+20 FROM t RETURNING *;
!  a
! ----
!  21
!  22
!  23
!  24
!  25
!  26
!  27
!  28
!  29
!  30
! (10 rows)
!
  WITH t AS (
  	SELECT a FROM y
  )
  UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
!  a
! ----
!  11
!  12
!  13
!  14
!  15
!  16
!  17
!  18
!  19
!  20
! (10 rows)
  WITH RECURSIVE t(a) AS (
  	SELECT 11
--- 747,762 ----
  )
  INSERT INTO y
  SELECT a+20 FROM t RETURNING *;
! ERROR:  relation "t" does not exist
! LINE 5: SELECT a+20 FROM t RETURNING *;
!                          ^
  WITH t AS (
  	SELECT a FROM y
  )
  UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
!  a
! ---
! (0 rows)

WITH RECURSIVE t(a) AS (
SELECT 11
***************
*** 785,803 ****
SELECT a+1 FROM t WHERE a < 50
)
DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
! a
! ----
! 11
! 12
! 13
! 14
! 15
! 16
! 17
! 18
! 19
! 20
! (10 rows)

  SELECT * FROM y;
   a
--- 764,772 ----
  	SELECT a+1 FROM t WHERE a < 50
  )
  DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
!  a
! ---
! (0 rows)

SELECT * FROM y;
a

======================================================================

hth,

Erik Rijkers

#24Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Erik Rijkers (#23)
1 attachment(s)
Re: [HACKERS] top-level DML under CTEs

On 2010-10-04 2:46 PM +0300, Erik Rijkers wrote:

(HEAD from git://git.postgresql.org/git/postgresql.git)

The patch applies only with error.
If that error is ignored, the regression 'with' test failes.
If that is also ignored, it runs.

This patch conflicted with Tom's WITH .. INSERT change. I tweaked that
patch just a bit and it now passes all regression tests so I can review
it. New version attached for documentation purposes.

Regards,
Marko Tiikkaja

Attachments:

dml.patchtext/plain; charset=iso-8859-1; name=dml.patch; x-mac-creator=0; x-mac-type=0Download
*** a/doc/src/sgml/ref/delete.sgml
--- b/doc/src/sgml/ref/delete.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
      [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
***************
*** 84,89 **** DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
--- 85,102 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><literal>ONLY</></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/insert.sgml
--- b/doc/src/sgml/ref/insert.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
      { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
      [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
***************
*** 85,90 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
--- 86,109 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+      <para>
+       It is possible that <literal>SELECT</literal> query also has
+       <literal>WITH</literal>.  In this case the two
+       <replaceable>with_query</replaceable> can be referred from
+       the <literal>SELECT</literal> query.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
***************
*** 129,135 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
      <listitem>
       <para>
        The corresponding <replaceable>column</replaceable> will be filled with
!       its default value.
       </para>
      </listitem>
     </varlistentry>
--- 148,155 ----
      <listitem>
       <para>
        The corresponding <replaceable>column</replaceable> will be filled with
!       its default value. This clause is allowed in a simple VALUES list
!       without additional (LIMIT, etc.) clauses.
       </para>
      </listitem>
     </varlistentry>
*** a/doc/src/sgml/ref/update.sgml
--- b/doc/src/sgml/ref/update.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
            ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
***************
*** 80,85 **** UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
--- 81,98 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***************
*** 2288,2293 **** _copyInsertStmt(InsertStmt *from)
--- 2288,2294 ----
  	COPY_NODE_FIELD(cols);
  	COPY_NODE_FIELD(selectStmt);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2301,2306 **** _copyDeleteStmt(DeleteStmt *from)
--- 2302,2308 ----
  	COPY_NODE_FIELD(usingClause);
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2315,2320 **** _copyUpdateStmt(UpdateStmt *from)
--- 2317,2323 ----
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(fromClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
*** a/src/backend/nodes/equalfuncs.c
--- b/src/backend/nodes/equalfuncs.c
***************
*** 890,895 **** _equalInsertStmt(InsertStmt *a, InsertStmt *b)
--- 890,896 ----
  	COMPARE_NODE_FIELD(cols);
  	COMPARE_NODE_FIELD(selectStmt);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 901,906 **** _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
--- 902,908 ----
  	COMPARE_NODE_FIELD(usingClause);
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 913,918 **** _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
--- 915,921 ----
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(fromClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
*** a/src/backend/parser/analyze.c
--- b/src/backend/parser/analyze.c
***************
*** 283,288 **** transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
--- 283,295 ----
  
  	qry->commandType = CMD_DELETE;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/* set up range table with just the result rel */
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
***************
*** 343,348 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
--- 350,362 ----
  	qry->commandType = CMD_INSERT;
  	pstate->p_is_insert = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/*
  	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
  	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
***************
*** 376,383 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		pstate->p_relnamespace = NIL;
  		sub_varnamespace = pstate->p_varnamespace;
  		pstate->p_varnamespace = NIL;
- 		/* There can't be any outer WITH to worry about */
- 		Assert(pstate->p_ctenamespace == NIL);
  	}
  	else
  	{
--- 390,395 ----
***************
*** 518,530 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		List	   *exprsLists = NIL;
  		int			sublist_length = -1;
  
- 		/* process the WITH clause */
- 		if (selectStmt->withClause)
- 		{
- 			qry->hasRecursive = selectStmt->withClause->recursive;
- 			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
- 		}
- 
  		foreach(lc, selectStmt->valuesLists)
  		{
  			List	   *sublist = (List *) lfirst(lc);
--- 530,535 ----
***************
*** 618,630 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  
  		Assert(list_length(valuesLists) == 1);
  
- 		/* process the WITH clause */
- 		if (selectStmt->withClause)
- 		{
- 			qry->hasRecursive = selectStmt->withClause->recursive;
- 			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
- 		}
- 
  		/* Do basic expression transformation (same as a ROW() expr) */
  		exprList = transformExpressionList(pstate,
  										   (List *) linitial(valuesLists));
--- 623,628 ----
***************
*** 1045,1051 **** transformValuesClause(ParseState *pstate, SelectStmt *stmt)
  			if (IsA(col, SetToDefault))
  				ereport(ERROR,
  						(errcode(ERRCODE_SYNTAX_ERROR),
! 						 errmsg("DEFAULT can only appear in a VALUES list within INSERT"),
  						 parser_errposition(pstate, exprLocation(col))));
  			colexprs[i] = lappend(colexprs[i], col);
  			i++;
--- 1043,1050 ----
  			if (IsA(col, SetToDefault))
  				ereport(ERROR,
  						(errcode(ERRCODE_SYNTAX_ERROR),
! 						 errmsg("DEFAULT can only appear in a simple VALUES list within INSERT"),
! 						 errhint("A simple VALUES list means it isn't modified by any of ORDER BY, OFFSET, LIMIT or WITH"),
  						 parser_errposition(pstate, exprLocation(col))));
  			colexprs[i] = lappend(colexprs[i], col);
  			i++;
***************
*** 1794,1799 **** transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
--- 1793,1805 ----
  	qry->commandType = CMD_UPDATE;
  	pstate->p_is_update = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
  										 true,
*** a/src/backend/parser/gram.y
--- b/src/backend/parser/gram.y
***************
*** 433,439 **** static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
--- 433,439 ----
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause opt_with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
***************
*** 7268,7278 **** DeallocateStmt: DEALLOCATE name
   *****************************************************************************/
  
  InsertStmt:
! 			INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$4->relation = $3;
! 					$4->returningList = $5;
! 					$$ = (Node *) $4;
  				}
  		;
  
--- 7268,7279 ----
   *****************************************************************************/
  
  InsertStmt:
! 			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$5->relation = $4;
! 					$5->returningList = $6;
! 					$5->withClause = $1;
! 					$$ = (Node *) $5;
  				}
  		;
  
***************
*** 7328,7341 **** returning_clause:
   *
   *****************************************************************************/
  
! DeleteStmt: DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $3;
! 					n->usingClause = $4;
! 					n->whereClause = $5;
! 					n->returningList = $6;
  					$$ = (Node *)n;
  				}
  		;
--- 7329,7343 ----
   *
   *****************************************************************************/
  
! DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $4;
! 					n->usingClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7390,7407 **** opt_nowait:	NOWAIT							{ $$ = TRUE; }
   *
   *****************************************************************************/
  
! UpdateStmt: UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $2;
! 					n->targetList = $4;
! 					n->fromClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
  					$$ = (Node *)n;
  				}
  		;
--- 7392,7410 ----
   *
   *****************************************************************************/
  
! UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $3;
! 					n->targetList = $5;
! 					n->fromClause = $6;
! 					n->whereClause = $7;
! 					n->returningList = $8;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7743,7748 **** common_table_expr:  name opt_name_list AS select_with_parens
--- 7746,7757 ----
  			}
  		;
  
+ opt_with_clause:
+ 		with_clause								{ $$ = $1; }
+ 		| /*EMPTY*/								{ $$ = NULL; }
+ 		;
+ 
+ 
  into_clause:
  			INTO OptTempTableName
  				{
*** a/src/backend/parser/parse_utilcmd.c
--- b/src/backend/parser/parse_utilcmd.c
***************
*** 1868,1873 **** transformRuleStmt(RuleStmt *stmt, const char *queryString,
--- 1868,1881 ----
  			}
  
  			/*
+ 			 * OLD/NEW is not allowed in CTE queries.
+ 			 */
+ 			if (checkCTEHasOldNew(sub_qry))
+ 				ereport(ERROR,
+ 						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ 						 errmsg("cannot refer to OLD/NEW in CTE query")));
+ 
+ 			/*
  			 * For efficiency's sake, add OLD to the rule action's jointree
  			 * only if it was actually referenced in the statement or qual.
  			 *
*** a/src/backend/rewrite/rewriteManip.c
--- b/src/backend/rewrite/rewriteManip.c
***************
*** 291,296 **** checkExprHasSubLink_walker(Node *node, void *context)
--- 291,320 ----
  	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
  }
  
+ /*
+  * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+  */
+ bool
+ checkCTEHasOldNew(Query *node)
+ {
+ 	ListCell   *l;
+ 
+ 	foreach (l, node->cteList)
+ 	{
+ 		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+ 		int		new_varno = PRS2_NEW_VARNO;
+ 		int		old_varno = PRS2_OLD_VARNO;
+ 
+ 		/* 1 == the top CTE */
+ 		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+ 			return true;
+ 
+ 		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+ 			return true;
+ 	}
+ 
+ 	return false;
+ }
  
  /*
   * OffsetVarNodes - adjust Vars when appending one query's RT to another
*** a/src/backend/utils/adt/ruleutils.c
--- b/src/backend/utils/adt/ruleutils.c
***************
*** 3345,3350 **** get_insert_query_def(Query *query, deparse_context *context)
--- 3345,3353 ----
  	ListCell   *l;
  	List	   *strippedexprs;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
  	 * a single RTE for the SELECT or VALUES.
***************
*** 3482,3487 **** get_update_query_def(Query *query, deparse_context *context)
--- 3485,3493 ----
  	RangeTblEntry *rte;
  	ListCell   *l;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with UPDATE relname SET
  	 */
***************
*** 3563,3568 **** get_delete_query_def(Query *query, deparse_context *context)
--- 3569,3577 ----
  	StringInfo	buf = context->buf;
  	RangeTblEntry *rte;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with DELETE FROM relname
  	 */
*** a/src/include/nodes/parsenodes.h
--- b/src/include/nodes/parsenodes.h
***************
*** 896,901 **** typedef struct InsertStmt
--- 896,902 ----
  	List	   *cols;			/* optional: names of the target columns */
  	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } InsertStmt;
  
  /* ----------------------
***************
*** 909,914 **** typedef struct DeleteStmt
--- 910,916 ----
  	List	   *usingClause;	/* optional using clause for more tables */
  	Node	   *whereClause;	/* qualifications */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } DeleteStmt;
  
  /* ----------------------
***************
*** 923,928 **** typedef struct UpdateStmt
--- 925,931 ----
  	Node	   *whereClause;	/* qualifications */
  	List	   *fromClause;		/* optional from clause for more tables */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } UpdateStmt;
  
  /* ----------------------
*** a/src/include/rewrite/rewriteManip.h
--- b/src/include/rewrite/rewriteManip.h
***************
*** 56,61 **** extern int	locate_windowfunc(Node *node);
--- 56,62 ----
  extern bool checkExprHasAggs(Node *node);
  extern bool checkExprHasWindowFuncs(Node *node);
  extern bool checkExprHasSubLink(Node *node);
+ extern bool checkCTEHasOldNew(Query *node);
  
  extern Node *replace_rte_variables(Node *node,
  					  int target_varno, int sublevels_up,
*** a/src/test/regress/expected/with.out
--- b/src/test/regress/expected/with.out
***************
*** 738,743 **** WITH RECURSIVE
--- 738,820 ----
  (54 rows)
  
  --
+ -- WITH on top of a DML statement
+ --
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+  a  
+ ----
+  21
+  22
+  23
+  24
+  25
+  26
+  27
+  28
+  29
+  30
+ (10 rows)
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ SELECT * FROM y;
+  a  
+ ----
+   1
+   2
+   3
+   4
+   5
+   6
+   7
+   8
+   9
+  10
+ (10 rows)
+ 
+ --
  -- error cases
  --
  -- INTERSECT
***************
*** 774,781 **** WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  ERROR:  recursive reference to query "x" must not appear within its non-recursive term
  LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                                ^
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
  -- LEFT JOIN
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
  	UNION ALL
--- 851,856 ----
***************
*** 912,917 **** ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
--- 987,997 ----
  LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                     ^
  HINT:  Cast the output of the non-recursive term to the correct type.
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ ERROR:  cannot refer to OLD/NEW in CTE query
  --
  -- test for bug #4902
  --
*** a/src/test/regress/sql/with.sql
--- b/src/test/regress/sql/with.sql
***************
*** 339,344 **** WITH RECURSIVE
--- 339,371 ----
   SELECT * FROM z;
  
  --
+ -- WITH on top of a DML statement
+ --
+ 
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ 
+ SELECT * FROM y;
+ 
+ --
  -- error cases
  --
  
***************
*** 364,372 **** WITH RECURSIVE x(n) AS (SELECT n FROM x)
  WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  	SELECT * FROM x;
  
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
- 
  -- LEFT JOIN
  
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
--- 391,396 ----
***************
*** 470,475 **** WITH RECURSIVE foo(i) AS
--- 494,504 ----
     SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
  SELECT * FROM foo;
  
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ 
  --
  -- test for bug #4902
  --
#25Robert Haas
robertmhaas@gmail.com
In reply to: Marko Tiikkaja (#24)
Re: [HACKERS] top-level DML under CTEs

On Mon, Oct 4, 2010 at 5:59 PM, Marko Tiikkaja
<marko.tiikkaja@cs.helsinki.fi> wrote:

On 2010-10-04 2:46 PM +0300, Erik Rijkers wrote:

(HEAD from git://git.postgresql.org/git/postgresql.git)

The patch applies only with error.
If that error is ignored, the regression 'with' test failes.
If that is also ignored, it runs.

This patch conflicted with Tom's WITH .. INSERT change.  I tweaked that
patch just a bit and it now passes all regression tests so I can review it.
 New version attached for documentation purposes.

Guys -

PLEASE redirect this conversation to -hackers!

Thanks,

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

#26Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Robert Haas (#25)
1 attachment(s)
Re: top-level DML under CTEs

(Oops, this didn't go to -HACKERS)

On 2010-10-04 2:46 PM +0300, Erik Rijkers wrote:

(HEAD from git://git.postgresql.org/git/postgresql.git)

The patch applies only with error.
If that error is ignored, the regression 'with' test failes.
If that is also ignored, it runs.

This patch conflicted with Tom's WITH .. INSERT change. I tweaked the
patch just a bit and it now passes all regression tests so I can review
it. New version attached for documentation purposes.

Regards,
Marko Tiikkaja

Attachments:

dml.patchtext/plain; charset=iso-8859-1; name=dml.patch; x-mac-creator=0; x-mac-type=0Download
*** a/doc/src/sgml/ref/delete.sgml
--- b/doc/src/sgml/ref/delete.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
      [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
***************
*** 84,89 **** DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
--- 85,102 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><literal>ONLY</></term>
      <listitem>
       <para>
*** a/doc/src/sgml/ref/insert.sgml
--- b/doc/src/sgml/ref/insert.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
      { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
      [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
***************
*** 85,90 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
--- 86,109 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+      <para>
+       It is possible that <literal>SELECT</literal> query also has
+       <literal>WITH</literal>.  In this case the two
+       <replaceable>with_query</replaceable> can be referred from
+       the <literal>SELECT</literal> query.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
***************
*** 129,135 **** INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
      <listitem>
       <para>
        The corresponding <replaceable>column</replaceable> will be filled with
!       its default value.
       </para>
      </listitem>
     </varlistentry>
--- 148,155 ----
      <listitem>
       <para>
        The corresponding <replaceable>column</replaceable> will be filled with
!       its default value. This clause is allowed in a simple VALUES list
!       without additional (LIMIT, etc.) clauses.
       </para>
      </listitem>
     </varlistentry>
*** a/doc/src/sgml/ref/update.sgml
--- b/doc/src/sgml/ref/update.sgml
***************
*** 21,26 **** PostgreSQL documentation
--- 21,27 ----
  
   <refsynopsisdiv>
  <synopsis>
+ [ WITH [ RECURSIVE ] with_query ]
  UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
      SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
            ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
***************
*** 80,85 **** UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
--- 81,98 ----
  
    <variablelist>
     <varlistentry>
+     <term><replaceable class="PARAMETER">with_query</replaceable></term>
+     <listitem>
+      <para>
+       The <literal>WITH</literal> clause allows you to specify one or more
+       subqueries that can be referenced by name in the primary query.
+       See <xref linkend="queries-with"> and <xref linkend="sql-select">
+       for details.
+      </para>
+     </listitem>
+    </varlistentry>
+ 
+    <varlistentry>
      <term><replaceable class="PARAMETER">table</replaceable></term>
      <listitem>
       <para>
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***************
*** 2288,2293 **** _copyInsertStmt(InsertStmt *from)
--- 2288,2294 ----
  	COPY_NODE_FIELD(cols);
  	COPY_NODE_FIELD(selectStmt);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2301,2306 **** _copyDeleteStmt(DeleteStmt *from)
--- 2302,2308 ----
  	COPY_NODE_FIELD(usingClause);
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
***************
*** 2315,2320 **** _copyUpdateStmt(UpdateStmt *from)
--- 2317,2323 ----
  	COPY_NODE_FIELD(whereClause);
  	COPY_NODE_FIELD(fromClause);
  	COPY_NODE_FIELD(returningList);
+ 	COPY_NODE_FIELD(withClause);
  
  	return newnode;
  }
*** a/src/backend/nodes/equalfuncs.c
--- b/src/backend/nodes/equalfuncs.c
***************
*** 890,895 **** _equalInsertStmt(InsertStmt *a, InsertStmt *b)
--- 890,896 ----
  	COMPARE_NODE_FIELD(cols);
  	COMPARE_NODE_FIELD(selectStmt);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 901,906 **** _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
--- 902,908 ----
  	COMPARE_NODE_FIELD(usingClause);
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
***************
*** 913,918 **** _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
--- 915,921 ----
  	COMPARE_NODE_FIELD(whereClause);
  	COMPARE_NODE_FIELD(fromClause);
  	COMPARE_NODE_FIELD(returningList);
+ 	COMPARE_NODE_FIELD(withClause);
  
  	return true;
  }
*** a/src/backend/parser/analyze.c
--- b/src/backend/parser/analyze.c
***************
*** 283,288 **** transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
--- 283,295 ----
  
  	qry->commandType = CMD_DELETE;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/* set up range table with just the result rel */
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
***************
*** 343,348 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
--- 350,362 ----
  	qry->commandType = CMD_INSERT;
  	pstate->p_is_insert = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	/*
  	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
  	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
***************
*** 376,383 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		pstate->p_relnamespace = NIL;
  		sub_varnamespace = pstate->p_varnamespace;
  		pstate->p_varnamespace = NIL;
- 		/* There can't be any outer WITH to worry about */
- 		Assert(pstate->p_ctenamespace == NIL);
  	}
  	else
  	{
--- 390,395 ----
***************
*** 518,530 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  		List	   *exprsLists = NIL;
  		int			sublist_length = -1;
  
- 		/* process the WITH clause */
- 		if (selectStmt->withClause)
- 		{
- 			qry->hasRecursive = selectStmt->withClause->recursive;
- 			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
- 		}
- 
  		foreach(lc, selectStmt->valuesLists)
  		{
  			List	   *sublist = (List *) lfirst(lc);
--- 530,535 ----
***************
*** 618,630 **** transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
  
  		Assert(list_length(valuesLists) == 1);
  
- 		/* process the WITH clause */
- 		if (selectStmt->withClause)
- 		{
- 			qry->hasRecursive = selectStmt->withClause->recursive;
- 			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
- 		}
- 
  		/* Do basic expression transformation (same as a ROW() expr) */
  		exprList = transformExpressionList(pstate,
  										   (List *) linitial(valuesLists));
--- 623,628 ----
***************
*** 1045,1051 **** transformValuesClause(ParseState *pstate, SelectStmt *stmt)
  			if (IsA(col, SetToDefault))
  				ereport(ERROR,
  						(errcode(ERRCODE_SYNTAX_ERROR),
! 						 errmsg("DEFAULT can only appear in a VALUES list within INSERT"),
  						 parser_errposition(pstate, exprLocation(col))));
  			colexprs[i] = lappend(colexprs[i], col);
  			i++;
--- 1043,1050 ----
  			if (IsA(col, SetToDefault))
  				ereport(ERROR,
  						(errcode(ERRCODE_SYNTAX_ERROR),
! 						 errmsg("DEFAULT can only appear in a simple VALUES list within INSERT"),
! 						 errhint("A simple VALUES list means it isn't modified by any of ORDER BY, OFFSET, LIMIT or WITH"),
  						 parser_errposition(pstate, exprLocation(col))));
  			colexprs[i] = lappend(colexprs[i], col);
  			i++;
***************
*** 1794,1799 **** transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
--- 1793,1805 ----
  	qry->commandType = CMD_UPDATE;
  	pstate->p_is_update = true;
  
+ 	/* process the WITH clause independently of all else */
+ 	if (stmt->withClause)
+ 	{
+ 		qry->hasRecursive = stmt->withClause->recursive;
+ 		qry->cteList = transformWithClause(pstate, stmt->withClause);
+ 	}
+ 
  	qry->resultRelation = setTargetTable(pstate, stmt->relation,
  								  interpretInhOption(stmt->relation->inhOpt),
  										 true,
*** a/src/backend/parser/gram.y
--- b/src/backend/parser/gram.y
***************
*** 433,439 **** static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
--- 433,439 ----
  %type <boolean> xml_whitespace_option
  
  %type <node> 	common_table_expr
! %type <with> 	with_clause opt_with_clause
  %type <list>	cte_list
  
  %type <list>	window_clause window_definition_list opt_partition_clause
***************
*** 7268,7278 **** DeallocateStmt: DEALLOCATE name
   *****************************************************************************/
  
  InsertStmt:
! 			INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$4->relation = $3;
! 					$4->returningList = $5;
! 					$$ = (Node *) $4;
  				}
  		;
  
--- 7268,7279 ----
   *****************************************************************************/
  
  InsertStmt:
! 			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
  				{
! 					$5->relation = $4;
! 					$5->returningList = $6;
! 					$5->withClause = $1;
! 					$$ = (Node *) $5;
  				}
  		;
  
***************
*** 7328,7341 **** returning_clause:
   *
   *****************************************************************************/
  
! DeleteStmt: DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $3;
! 					n->usingClause = $4;
! 					n->whereClause = $5;
! 					n->returningList = $6;
  					$$ = (Node *)n;
  				}
  		;
--- 7329,7343 ----
   *
   *****************************************************************************/
  
! DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
  			using_clause where_or_current_clause returning_clause
  				{
  					DeleteStmt *n = makeNode(DeleteStmt);
! 					n->relation = $4;
! 					n->usingClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7390,7407 **** opt_nowait:	NOWAIT							{ $$ = TRUE; }
   *
   *****************************************************************************/
  
! UpdateStmt: UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $2;
! 					n->targetList = $4;
! 					n->fromClause = $5;
! 					n->whereClause = $6;
! 					n->returningList = $7;
  					$$ = (Node *)n;
  				}
  		;
--- 7392,7410 ----
   *
   *****************************************************************************/
  
! UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
  			SET set_clause_list
  			from_clause
  			where_or_current_clause
  			returning_clause
  				{
  					UpdateStmt *n = makeNode(UpdateStmt);
! 					n->relation = $3;
! 					n->targetList = $5;
! 					n->fromClause = $6;
! 					n->whereClause = $7;
! 					n->returningList = $8;
! 					n->withClause = $1;
  					$$ = (Node *)n;
  				}
  		;
***************
*** 7743,7748 **** common_table_expr:  name opt_name_list AS select_with_parens
--- 7746,7757 ----
  			}
  		;
  
+ opt_with_clause:
+ 		with_clause								{ $$ = $1; }
+ 		| /*EMPTY*/								{ $$ = NULL; }
+ 		;
+ 
+ 
  into_clause:
  			INTO OptTempTableName
  				{
*** a/src/backend/parser/parse_utilcmd.c
--- b/src/backend/parser/parse_utilcmd.c
***************
*** 1868,1873 **** transformRuleStmt(RuleStmt *stmt, const char *queryString,
--- 1868,1881 ----
  			}
  
  			/*
+ 			 * OLD/NEW is not allowed in CTE queries.
+ 			 */
+ 			if (checkCTEHasOldNew(sub_qry))
+ 				ereport(ERROR,
+ 						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ 						 errmsg("cannot refer to OLD/NEW in CTE query")));
+ 
+ 			/*
  			 * For efficiency's sake, add OLD to the rule action's jointree
  			 * only if it was actually referenced in the statement or qual.
  			 *
*** a/src/backend/rewrite/rewriteManip.c
--- b/src/backend/rewrite/rewriteManip.c
***************
*** 291,296 **** checkExprHasSubLink_walker(Node *node, void *context)
--- 291,320 ----
  	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
  }
  
+ /*
+  * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+  */
+ bool
+ checkCTEHasOldNew(Query *node)
+ {
+ 	ListCell   *l;
+ 
+ 	foreach (l, node->cteList)
+ 	{
+ 		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+ 		int		new_varno = PRS2_NEW_VARNO;
+ 		int		old_varno = PRS2_OLD_VARNO;
+ 
+ 		/* 1 == the top CTE */
+ 		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+ 			return true;
+ 
+ 		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+ 			return true;
+ 	}
+ 
+ 	return false;
+ }
  
  /*
   * OffsetVarNodes - adjust Vars when appending one query's RT to another
*** a/src/backend/utils/adt/ruleutils.c
--- b/src/backend/utils/adt/ruleutils.c
***************
*** 3345,3350 **** get_insert_query_def(Query *query, deparse_context *context)
--- 3345,3353 ----
  	ListCell   *l;
  	List	   *strippedexprs;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
  	 * a single RTE for the SELECT or VALUES.
***************
*** 3482,3487 **** get_update_query_def(Query *query, deparse_context *context)
--- 3485,3493 ----
  	RangeTblEntry *rte;
  	ListCell   *l;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with UPDATE relname SET
  	 */
***************
*** 3563,3568 **** get_delete_query_def(Query *query, deparse_context *context)
--- 3569,3577 ----
  	StringInfo	buf = context->buf;
  	RangeTblEntry *rte;
  
+ 	/* Insert the WITH clause if given */
+ 	get_with_clause(query, context);
+ 
  	/*
  	 * Start the query with DELETE FROM relname
  	 */
*** a/src/include/nodes/parsenodes.h
--- b/src/include/nodes/parsenodes.h
***************
*** 896,901 **** typedef struct InsertStmt
--- 896,902 ----
  	List	   *cols;			/* optional: names of the target columns */
  	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } InsertStmt;
  
  /* ----------------------
***************
*** 909,914 **** typedef struct DeleteStmt
--- 910,916 ----
  	List	   *usingClause;	/* optional using clause for more tables */
  	Node	   *whereClause;	/* qualifications */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } DeleteStmt;
  
  /* ----------------------
***************
*** 923,928 **** typedef struct UpdateStmt
--- 925,931 ----
  	Node	   *whereClause;	/* qualifications */
  	List	   *fromClause;		/* optional from clause for more tables */
  	List	   *returningList;	/* list of expressions to return */
+ 	WithClause *withClause;		/* WITH clause */
  } UpdateStmt;
  
  /* ----------------------
*** a/src/include/rewrite/rewriteManip.h
--- b/src/include/rewrite/rewriteManip.h
***************
*** 56,61 **** extern int	locate_windowfunc(Node *node);
--- 56,62 ----
  extern bool checkExprHasAggs(Node *node);
  extern bool checkExprHasWindowFuncs(Node *node);
  extern bool checkExprHasSubLink(Node *node);
+ extern bool checkCTEHasOldNew(Query *node);
  
  extern Node *replace_rte_variables(Node *node,
  					  int target_varno, int sublevels_up,
*** a/src/test/regress/expected/with.out
--- b/src/test/regress/expected/with.out
***************
*** 738,743 **** WITH RECURSIVE
--- 738,820 ----
  (54 rows)
  
  --
+ -- WITH on top of a DML statement
+ --
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+  a  
+ ----
+  21
+  22
+  23
+  24
+  25
+  26
+  27
+  28
+  29
+  30
+ (10 rows)
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+  a  
+ ----
+  11
+  12
+  13
+  14
+  15
+  16
+  17
+  18
+  19
+  20
+ (10 rows)
+ 
+ SELECT * FROM y;
+  a  
+ ----
+   1
+   2
+   3
+   4
+   5
+   6
+   7
+   8
+   9
+  10
+ (10 rows)
+ 
+ --
  -- error cases
  --
  -- INTERSECT
***************
*** 774,781 **** WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  ERROR:  recursive reference to query "x" must not appear within its non-recursive term
  LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                                ^
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
  -- LEFT JOIN
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
  	UNION ALL
--- 851,856 ----
***************
*** 912,917 **** ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
--- 987,997 ----
  LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                     ^
  HINT:  Cast the output of the non-recursive term to the correct type.
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ ERROR:  cannot refer to OLD/NEW in CTE query
  --
  -- test for bug #4902
  --
*** a/src/test/regress/sql/with.sql
--- b/src/test/regress/sql/with.sql
***************
*** 339,344 **** WITH RECURSIVE
--- 339,371 ----
   SELECT * FROM z;
  
  --
+ -- WITH on top of a DML statement
+ --
+ 
+ CREATE TEMPORARY TABLE y (a INTEGER);
+ INSERT INTO y SELECT generate_series(1, 10);
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ INSERT INTO y
+ SELECT a+20 FROM t RETURNING *;
+ 
+ WITH t AS (
+ 	SELECT a FROM y
+ )
+ UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ 
+ WITH RECURSIVE t(a) AS (
+ 	SELECT 11
+ 	UNION ALL
+ 	SELECT a+1 FROM t WHERE a < 50
+ )
+ DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ 
+ SELECT * FROM y;
+ 
+ --
  -- error cases
  --
  
***************
*** 364,372 **** WITH RECURSIVE x(n) AS (SELECT n FROM x)
  WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
  	SELECT * FROM x;
  
- CREATE TEMPORARY TABLE y (a INTEGER);
- INSERT INTO y SELECT generate_series(1, 10);
- 
  -- LEFT JOIN
  
  WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
--- 391,396 ----
***************
*** 470,475 **** WITH RECURSIVE foo(i) AS
--- 494,504 ----
     SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
  SELECT * FROM foo;
  
+ -- disallow OLD/NEW reference in CTE
+ CREATE TEMPORARY TABLE x (n integer);
+ CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+     WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ 
  --
  -- test for bug #4902
  --

#27Hitoshi Harada
umi.tanuki@gmail.com
In reply to: Marko Tiikkaja (#26)
1 attachment(s)
Re: top-level DML under CTEs

2010/10/5 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

(Oops, this didn't go to -HACKERS)

On 2010-10-04 2:46 PM +0300, Erik Rijkers wrote:

(HEAD from git://git.postgresql.org/git/postgresql.git)

The patch applies only with error.
If that error is ignored, the regression 'with' test failes.
If that is also ignored, it runs.

This patch conflicted with Tom's WITH .. INSERT change.  I tweaked the
patch just a bit and it now passes all regression tests so I can review
it.  New version attached for documentation purposes.

Thank you, I didn't notice that commit. In your last patch, the
snippet to add errhint() and ref/insert sgml is unnecessary since it
was for INSERT ... VALUES fix.

Regards,

--
Hitoshi Harada

Attachments:

toplevel-dml-cte.20101005.diffapplication/octet-stream; name=toplevel-dml-cte.20101005.diffDownload
diff --git a/doc/src/sgml/ref/delete.sgml b/doc/src/sgml/ref/delete.sgml
index c87f35c..45a1643 100644
--- a/doc/src/sgml/ref/delete.sgml
+++ b/doc/src/sgml/ref/delete.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
     [ USING <replaceable class="PARAMETER">using_list</replaceable> ]
     [ WHERE <replaceable class="PARAMETER">condition</replaceable> | WHERE CURRENT OF <replaceable class="PARAMETER">cursor_name</replaceable> ]
@@ -84,6 +85,18 @@ DELETE FROM [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ]
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><literal>ONLY</></term>
     <listitem>
      <para>
diff --git a/doc/src/sgml/ref/insert.sgml b/doc/src/sgml/ref/insert.sgml
index 6d17ef0..73685b9 100644
--- a/doc/src/sgml/ref/insert.sgml
+++ b/doc/src/sgml/ref/insert.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) ]
     { DEFAULT VALUES | VALUES ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) [, ...] | <replaceable class="PARAMETER">query</replaceable> }
     [ RETURNING * | <replaceable class="parameter">output_expression</replaceable> [ [ AS ] <replaceable class="parameter">output_name</replaceable> ] [, ...] ]
@@ -85,6 +86,24 @@ INSERT INTO <replaceable class="PARAMETER">table</replaceable> [ ( <replaceable
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+     <para>
+      It is possible that <literal>SELECT</literal> query also has
+      <literal>WITH</literal>.  In this case the two
+      <replaceable>with_query</replaceable> can be referred from
+      the <literal>SELECT</literal> query.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><replaceable class="PARAMETER">table</replaceable></term>
     <listitem>
      <para>
diff --git a/doc/src/sgml/ref/update.sgml b/doc/src/sgml/ref/update.sgml
index c897634..08fb9a0 100644
--- a/doc/src/sgml/ref/update.sgml
+++ b/doc/src/sgml/ref/update.sgml
@@ -21,6 +21,7 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
+[ WITH [ RECURSIVE ] with_query ]
 UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <replaceable class="parameter">alias</replaceable> ]
     SET { <replaceable class="PARAMETER">column</replaceable> = { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } |
           ( <replaceable class="PARAMETER">column</replaceable> [, ...] ) = ( { <replaceable class="PARAMETER">expression</replaceable> | DEFAULT } [, ...] ) } [, ...]
@@ -80,6 +81,18 @@ UPDATE [ ONLY ] <replaceable class="PARAMETER">table</replaceable> [ [ AS ] <rep
 
   <variablelist>
    <varlistentry>
+    <term><replaceable class="PARAMETER">with_query</replaceable></term>
+    <listitem>
+     <para>
+      The <literal>WITH</literal> clause allows you to specify one or more
+      subqueries that can be referenced by name in the primary query.
+      See <xref linkend="queries-with"> and <xref linkend="sql-select">
+      for details.
+     </para>
+    </listitem>
+   </varlistentry>
+
+   <varlistentry>
     <term><replaceable class="PARAMETER">table</replaceable></term>
     <listitem>
      <para>
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 5bd0ef0..18b0b90 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2288,6 +2288,7 @@ _copyInsertStmt(InsertStmt *from)
 	COPY_NODE_FIELD(cols);
 	COPY_NODE_FIELD(selectStmt);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
@@ -2301,6 +2302,7 @@ _copyDeleteStmt(DeleteStmt *from)
 	COPY_NODE_FIELD(usingClause);
 	COPY_NODE_FIELD(whereClause);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
@@ -2315,6 +2317,7 @@ _copyUpdateStmt(UpdateStmt *from)
 	COPY_NODE_FIELD(whereClause);
 	COPY_NODE_FIELD(fromClause);
 	COPY_NODE_FIELD(returningList);
+	COPY_NODE_FIELD(withClause);
 
 	return newnode;
 }
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index c7dd42d..0ea3a31 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -890,6 +890,7 @@ _equalInsertStmt(InsertStmt *a, InsertStmt *b)
 	COMPARE_NODE_FIELD(cols);
 	COMPARE_NODE_FIELD(selectStmt);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
@@ -901,6 +902,7 @@ _equalDeleteStmt(DeleteStmt *a, DeleteStmt *b)
 	COMPARE_NODE_FIELD(usingClause);
 	COMPARE_NODE_FIELD(whereClause);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
@@ -913,6 +915,7 @@ _equalUpdateStmt(UpdateStmt *a, UpdateStmt *b)
 	COMPARE_NODE_FIELD(whereClause);
 	COMPARE_NODE_FIELD(fromClause);
 	COMPARE_NODE_FIELD(returningList);
+	COMPARE_NODE_FIELD(withClause);
 
 	return true;
 }
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 21342e8..af7be94 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -283,6 +283,13 @@ transformDeleteStmt(ParseState *pstate, DeleteStmt *stmt)
 
 	qry->commandType = CMD_DELETE;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	/* set up range table with just the result rel */
 	qry->resultRelation = setTargetTable(pstate, stmt->relation,
 								  interpretInhOption(stmt->relation->inhOpt),
@@ -343,6 +350,13 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 	qry->commandType = CMD_INSERT;
 	pstate->p_is_insert = true;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	/*
 	 * We have three cases to deal with: DEFAULT VALUES (selectStmt == NULL),
 	 * VALUES list, or general SELECT input.  We special-case VALUES, both for
@@ -376,8 +390,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 		pstate->p_relnamespace = NIL;
 		sub_varnamespace = pstate->p_varnamespace;
 		pstate->p_varnamespace = NIL;
-		/* There can't be any outer WITH to worry about */
-		Assert(pstate->p_ctenamespace == NIL);
 	}
 	else
 	{
@@ -518,13 +530,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 		List	   *exprsLists = NIL;
 		int			sublist_length = -1;
 
-		/* process the WITH clause */
-		if (selectStmt->withClause)
-		{
-			qry->hasRecursive = selectStmt->withClause->recursive;
-			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
-		}
-
 		foreach(lc, selectStmt->valuesLists)
 		{
 			List	   *sublist = (List *) lfirst(lc);
@@ -618,13 +623,6 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt)
 
 		Assert(list_length(valuesLists) == 1);
 
-		/* process the WITH clause */
-		if (selectStmt->withClause)
-		{
-			qry->hasRecursive = selectStmt->withClause->recursive;
-			qry->cteList = transformWithClause(pstate, selectStmt->withClause);
-		}
-
 		/* Do basic expression transformation (same as a ROW() expr) */
 		exprList = transformExpressionList(pstate,
 										   (List *) linitial(valuesLists));
@@ -1794,6 +1792,13 @@ transformUpdateStmt(ParseState *pstate, UpdateStmt *stmt)
 	qry->commandType = CMD_UPDATE;
 	pstate->p_is_update = true;
 
+	/* process the WITH clause independently of all else */
+	if (stmt->withClause)
+	{
+		qry->hasRecursive = stmt->withClause->recursive;
+		qry->cteList = transformWithClause(pstate, stmt->withClause);
+	}
+
 	qry->resultRelation = setTargetTable(pstate, stmt->relation,
 								  interpretInhOption(stmt->relation->inhOpt),
 										 true,
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4054cb1..401c001 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -433,7 +433,7 @@ static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_
 %type <boolean> xml_whitespace_option
 
 %type <node> 	common_table_expr
-%type <with> 	with_clause
+%type <with> 	with_clause opt_with_clause
 %type <list>	cte_list
 
 %type <list>	window_clause window_definition_list opt_partition_clause
@@ -7268,11 +7268,12 @@ DeallocateStmt: DEALLOCATE name
  *****************************************************************************/
 
 InsertStmt:
-			INSERT INTO qualified_name insert_rest returning_clause
+			opt_with_clause INSERT INTO qualified_name insert_rest returning_clause
 				{
-					$4->relation = $3;
-					$4->returningList = $5;
-					$$ = (Node *) $4;
+					$5->relation = $4;
+					$5->returningList = $6;
+					$5->withClause = $1;
+					$$ = (Node *) $5;
 				}
 		;
 
@@ -7328,14 +7329,15 @@ returning_clause:
  *
  *****************************************************************************/
 
-DeleteStmt: DELETE_P FROM relation_expr_opt_alias
+DeleteStmt: opt_with_clause DELETE_P FROM relation_expr_opt_alias
 			using_clause where_or_current_clause returning_clause
 				{
 					DeleteStmt *n = makeNode(DeleteStmt);
-					n->relation = $3;
-					n->usingClause = $4;
-					n->whereClause = $5;
-					n->returningList = $6;
+					n->relation = $4;
+					n->usingClause = $5;
+					n->whereClause = $6;
+					n->returningList = $7;
+					n->withClause = $1;
 					$$ = (Node *)n;
 				}
 		;
@@ -7390,18 +7392,19 @@ opt_nowait:	NOWAIT							{ $$ = TRUE; }
  *
  *****************************************************************************/
 
-UpdateStmt: UPDATE relation_expr_opt_alias
+UpdateStmt: opt_with_clause UPDATE relation_expr_opt_alias
 			SET set_clause_list
 			from_clause
 			where_or_current_clause
 			returning_clause
 				{
 					UpdateStmt *n = makeNode(UpdateStmt);
-					n->relation = $2;
-					n->targetList = $4;
-					n->fromClause = $5;
-					n->whereClause = $6;
-					n->returningList = $7;
+					n->relation = $3;
+					n->targetList = $5;
+					n->fromClause = $6;
+					n->whereClause = $7;
+					n->returningList = $8;
+					n->withClause = $1;
 					$$ = (Node *)n;
 				}
 		;
@@ -7743,6 +7746,12 @@ common_table_expr:  name opt_name_list AS select_with_parens
 			}
 		;
 
+opt_with_clause:
+		with_clause								{ $$ = $1; }
+		| /*EMPTY*/								{ $$ = NULL; }
+		;
+
+
 into_clause:
 			INTO OptTempTableName
 				{
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 37ca331..663754a 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -1868,6 +1868,14 @@ transformRuleStmt(RuleStmt *stmt, const char *queryString,
 			}
 
 			/*
+			 * OLD/NEW is not allowed in CTE queries.
+			 */
+			if (checkCTEHasOldNew(sub_qry))
+				ereport(ERROR,
+						(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+						 errmsg("cannot refer to OLD/NEW in CTE query")));
+
+			/*
 			 * For efficiency's sake, add OLD to the rule action's jointree
 			 * only if it was actually referenced in the statement or qual.
 			 *
diff --git a/src/backend/rewrite/rewriteManip.c b/src/backend/rewrite/rewriteManip.c
index 5db2522..ca6d494 100644
--- a/src/backend/rewrite/rewriteManip.c
+++ b/src/backend/rewrite/rewriteManip.c
@@ -291,6 +291,30 @@ checkExprHasSubLink_walker(Node *node, void *context)
 	return expression_tree_walker(node, checkExprHasSubLink_walker, context);
 }
 
+/*
+ * checkCTEHasOldNew - check if OLD/NEW is referred in CTE queries.
+ */
+bool
+checkCTEHasOldNew(Query *node)
+{
+	ListCell   *l;
+
+	foreach (l, node->cteList)
+	{
+		CommonTableExpr	   *cte = (CommonTableExpr *) lfirst(l);
+		int		new_varno = PRS2_NEW_VARNO;
+		int		old_varno = PRS2_OLD_VARNO;
+
+		/* 1 == the top CTE */
+		if (rangeTableEntry_used(cte->ctequery, new_varno, 1))
+			return true;
+
+		if (rangeTableEntry_used(cte->ctequery, old_varno, 1))
+			return true;
+	}
+
+	return false;
+}
 
 /*
  * OffsetVarNodes - adjust Vars when appending one query's RT to another
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 578b9ce..cec5e19 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -3345,6 +3345,9 @@ get_insert_query_def(Query *query, deparse_context *context)
 	ListCell   *l;
 	List	   *strippedexprs;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * If it's an INSERT ... SELECT or VALUES (...), (...), ... there will be
 	 * a single RTE for the SELECT or VALUES.
@@ -3482,6 +3485,9 @@ get_update_query_def(Query *query, deparse_context *context)
 	RangeTblEntry *rte;
 	ListCell   *l;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * Start the query with UPDATE relname SET
 	 */
@@ -3563,6 +3569,9 @@ get_delete_query_def(Query *query, deparse_context *context)
 	StringInfo	buf = context->buf;
 	RangeTblEntry *rte;
 
+	/* Insert the WITH clause if given */
+	get_with_clause(query, context);
+
 	/*
 	 * Start the query with DELETE FROM relname
 	 */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index b2f0fef..d93d759 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -896,6 +896,7 @@ typedef struct InsertStmt
 	List	   *cols;			/* optional: names of the target columns */
 	Node	   *selectStmt;		/* the source SELECT/VALUES, or NULL */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } InsertStmt;
 
 /* ----------------------
@@ -909,6 +910,7 @@ typedef struct DeleteStmt
 	List	   *usingClause;	/* optional using clause for more tables */
 	Node	   *whereClause;	/* qualifications */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } DeleteStmt;
 
 /* ----------------------
@@ -923,6 +925,7 @@ typedef struct UpdateStmt
 	Node	   *whereClause;	/* qualifications */
 	List	   *fromClause;		/* optional from clause for more tables */
 	List	   *returningList;	/* list of expressions to return */
+	WithClause *withClause;		/* WITH clause */
 } UpdateStmt;
 
 /* ----------------------
diff --git a/src/include/rewrite/rewriteManip.h b/src/include/rewrite/rewriteManip.h
index 8daea6e..7a1366b 100644
--- a/src/include/rewrite/rewriteManip.h
+++ b/src/include/rewrite/rewriteManip.h
@@ -56,6 +56,7 @@ extern int	locate_windowfunc(Node *node);
 extern bool checkExprHasAggs(Node *node);
 extern bool checkExprHasWindowFuncs(Node *node);
 extern bool checkExprHasSubLink(Node *node);
+extern bool checkCTEHasOldNew(Query *node);
 
 extern Node *replace_rte_variables(Node *node,
 					  int target_varno, int sublevels_up,
diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out
index e46ed78..a50148b 100644
--- a/src/test/regress/expected/with.out
+++ b/src/test/regress/expected/with.out
@@ -738,6 +738,83 @@ WITH RECURSIVE
 (54 rows)
 
 --
+-- WITH on top of a DML statement
+--
+CREATE TEMPORARY TABLE y (a INTEGER);
+INSERT INTO y SELECT generate_series(1, 10);
+WITH t AS (
+	SELECT a FROM y
+)
+INSERT INTO y
+SELECT a+20 FROM t RETURNING *;
+ a  
+----
+ 21
+ 22
+ 23
+ 24
+ 25
+ 26
+ 27
+ 28
+ 29
+ 30
+(10 rows)
+
+WITH t AS (
+	SELECT a FROM y
+)
+UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+ a  
+----
+ 11
+ 12
+ 13
+ 14
+ 15
+ 16
+ 17
+ 18
+ 19
+ 20
+(10 rows)
+
+WITH RECURSIVE t(a) AS (
+	SELECT 11
+	UNION ALL
+	SELECT a+1 FROM t WHERE a < 50
+)
+DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+ a  
+----
+ 11
+ 12
+ 13
+ 14
+ 15
+ 16
+ 17
+ 18
+ 19
+ 20
+(10 rows)
+
+SELECT * FROM y;
+ a  
+----
+  1
+  2
+  3
+  4
+  5
+  6
+  7
+  8
+  9
+ 10
+(10 rows)
+
+--
 -- error cases
 --
 -- INTERSECT
@@ -774,8 +851,6 @@ WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
 ERROR:  recursive reference to query "x" must not appear within its non-recursive term
 LINE 1: WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
                                               ^
-CREATE TEMPORARY TABLE y (a INTEGER);
-INSERT INTO y SELECT generate_series(1, 10);
 -- LEFT JOIN
 WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
 	UNION ALL
@@ -912,6 +987,11 @@ ERROR:  recursive query "foo" column 1 has type numeric(3,0) in non-recursive te
 LINE 2:    (SELECT i::numeric(3,0) FROM (VALUES(1),(2)) t(i)
                    ^
 HINT:  Cast the output of the non-recursive term to the correct type.
+-- disallow OLD/NEW reference in CTE
+CREATE TEMPORARY TABLE x (n integer);
+CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+    WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+ERROR:  cannot refer to OLD/NEW in CTE query
 --
 -- test for bug #4902
 --
diff --git a/src/test/regress/sql/with.sql b/src/test/regress/sql/with.sql
index 2cbaa42..7fc70c0 100644
--- a/src/test/regress/sql/with.sql
+++ b/src/test/regress/sql/with.sql
@@ -339,6 +339,33 @@ WITH RECURSIVE
  SELECT * FROM z;
 
 --
+-- WITH on top of a DML statement
+--
+
+CREATE TEMPORARY TABLE y (a INTEGER);
+INSERT INTO y SELECT generate_series(1, 10);
+
+WITH t AS (
+	SELECT a FROM y
+)
+INSERT INTO y
+SELECT a+20 FROM t RETURNING *;
+
+WITH t AS (
+	SELECT a FROM y
+)
+UPDATE y SET a = y.a-10 FROM t WHERE y.a > 20 AND t.a = y.a RETURNING y.a;
+
+WITH RECURSIVE t(a) AS (
+	SELECT 11
+	UNION ALL
+	SELECT a+1 FROM t WHERE a < 50
+)
+DELETE FROM y USING t WHERE t.a = y.a RETURNING y.a;
+
+SELECT * FROM y;
+
+--
 -- error cases
 --
 
@@ -364,9 +391,6 @@ WITH RECURSIVE x(n) AS (SELECT n FROM x)
 WITH RECURSIVE x(n) AS (SELECT n FROM x UNION ALL SELECT 1)
 	SELECT * FROM x;
 
-CREATE TEMPORARY TABLE y (a INTEGER);
-INSERT INTO y SELECT generate_series(1, 10);
-
 -- LEFT JOIN
 
 WITH RECURSIVE x(n) AS (SELECT a FROM y WHERE a = 1
@@ -470,6 +494,11 @@ WITH RECURSIVE foo(i) AS
    SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
 SELECT * FROM foo;
 
+-- disallow OLD/NEW reference in CTE
+CREATE TEMPORARY TABLE x (n integer);
+CREATE RULE r2 AS ON UPDATE TO x DO INSTEAD
+    WITH t AS (SELECT OLD.*) UPDATE y SET a = t.n FROM t;
+
 --
 -- test for bug #4902
 --
#28Marko Tiikkaja
marko.tiikkaja@cs.helsinki.fi
In reply to: Hitoshi Harada (#27)
Re: top-level DML under CTEs

On 2010-10-05 3:37 PM +0300, Hitoshi Harada wrote:

2010/10/5 Marko Tiikkaja<marko.tiikkaja@cs.helsinki.fi>:

This patch conflicted with Tom's WITH .. INSERT change. I tweaked the
patch just a bit and it now passes all regression tests so I can review
it. New version attached for documentation purposes.

Thank you, I didn't notice that commit. In your last patch, the
snippet to add errhint() and ref/insert sgml is unnecessary since it
was for INSERT ... VALUES fix.

The patch seems to work for all cases I can think of.

I'm going to mark this one ready for committer.

Regards,
Marko Tiikkaja

#29Tom Lane
tgl@sss.pgh.pa.us
In reply to: Hitoshi Harada (#27)
Re: top-level DML under CTEs

Hitoshi Harada <umi.tanuki@gmail.com> writes:

2010/10/5 Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi>:

This patch conflicted with Tom's WITH .. INSERT change. �I tweaked the
patch just a bit and it now passes all regression tests so I can review
it. �New version attached for documentation purposes.

Thank you, I didn't notice that commit. In your last patch, the
snippet to add errhint() and ref/insert sgml is unnecessary since it
was for INSERT ... VALUES fix.

Committed with minor fixes (mostly documentation improvements).

regards, tom lane