Refactor parse analysis of EXECUTE command
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt(). This makes EXECUTE behave more like other utility
commands.
Effects are that error messages can have position information (see
regression test case), and it allows using external parameters in the
arguments of the EXECUTE command.
I had previously inquired about this in [0]/messages/by-id/ed2767e5-c506-048d-8ddf-280ecbc9e1b7@2ndquadrant.com and some vague concerns were
raised. I haven't dug very deep on this, but I figure with an actual
patch it might be easier to review and figure out if there are any problems.
[0]: /messages/by-id/ed2767e5-c506-048d-8ddf-280ecbc9e1b7@2ndquadrant.com
/messages/by-id/ed2767e5-c506-048d-8ddf-280ecbc9e1b7@2ndquadrant.com
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt().
Uhmm ... no actual patch attached?
regards, tom lane
On 2019-11-02 16:00, Tom Lane wrote:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt().Uhmm ... no actual patch attached?
Oops, here it is.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Refactor-parse-analysis-of-EXECUTE-command.patchtext/plain; charset=UTF-8; name=0001-Refactor-parse-analysis-of-EXECUTE-command.patch; x-mac-creator=0; x-mac-type=0Download
From 7c5bc30a02ec34646c8e49af1499fe4113bc9e5e Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Thu, 31 Oct 2019 09:54:07 +0100
Subject: [PATCH] Refactor parse analysis of EXECUTE command
Move the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt(). This makes EXECUTE behave more like other utility
commands. It also allows error messages to have position information,
and it allows using external parameters in the arguments of the
EXECUTE command.
---
src/backend/commands/createas.c | 2 +-
src/backend/commands/prepare.c | 72 ++--------------------
src/backend/parser/analyze.c | 89 +++++++++++++++++++++++++++
src/backend/tcop/utility.c | 2 +-
src/include/commands/prepare.h | 2 +-
src/test/regress/expected/prepare.out | 25 ++++++++
src/test/regress/sql/prepare.sql | 21 +++++++
7 files changed, 143 insertions(+), 70 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index b7d220699f..e4244f84e2 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -271,7 +271,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
ExecuteStmt *estmt = castNode(ExecuteStmt, query->utilityStmt);
Assert(!is_matview); /* excluded by syntax */
- ExecuteQuery(estmt, into, queryString, params, dest, completionTag);
+ ExecuteQuery(estmt, into, params, dest, completionTag);
/* get object address that intorel_startup saved for us */
address = ((DR_intorel *) dest)->reladdr;
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 7e0a041fab..0aba6a7b00 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -47,7 +47,7 @@ static HTAB *prepared_queries = NULL;
static void InitQueryHashTable(void);
static ParamListInfo EvaluateParams(PreparedStatement *pstmt, List *params,
- const char *queryString, EState *estate);
+ EState *estate);
static Datum build_regtype_array(Oid *param_types, int num_params);
/*
@@ -189,16 +189,10 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString,
* indicated by passing a non-null intoClause. The DestReceiver is already
* set up correctly for CREATE TABLE AS, but we still have to make a few
* other adjustments here.
- *
- * Note: this is one of very few places in the code that needs to deal with
- * two query strings at once. The passed-in queryString is that of the
- * EXECUTE, which we might need for error reporting while processing the
- * parameter expressions. The query_string that we copy from the plan
- * source is that of the original PREPARE.
*/
void
ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
- const char *queryString, ParamListInfo params,
+ ParamListInfo params,
DestReceiver *dest, char *completionTag)
{
PreparedStatement *entry;
@@ -229,8 +223,7 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
*/
estate = CreateExecutorState();
estate->es_param_list_info = params;
- paramLI = EvaluateParams(entry, stmt->params,
- queryString, estate);
+ paramLI = EvaluateParams(entry, stmt->params, estate);
}
/* Create a new portal to run the query in */
@@ -316,7 +309,6 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
*
* pstmt: statement we are getting parameters for.
* params: list of given parameter expressions (raw parser output!)
- * queryString: source text for error messages.
* estate: executor state to use.
*
* Returns a filled-in ParamListInfo -- this can later be passed to
@@ -324,72 +316,19 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
* during query execution.
*/
static ParamListInfo
-EvaluateParams(PreparedStatement *pstmt, List *params,
- const char *queryString, EState *estate)
+EvaluateParams(PreparedStatement *pstmt, List *params, EState *estate)
{
Oid *param_types = pstmt->plansource->param_types;
int num_params = pstmt->plansource->num_params;
- int nparams = list_length(params);
- ParseState *pstate;
ParamListInfo paramLI;
List *exprstates;
ListCell *l;
int i;
- if (nparams != num_params)
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("wrong number of parameters for prepared statement \"%s\"",
- pstmt->stmt_name),
- errdetail("Expected %d parameters but got %d.",
- num_params, nparams)));
-
/* Quick exit if no parameters */
if (num_params == 0)
return NULL;
- /*
- * We have to run parse analysis for the expressions. Since the parser is
- * not cool about scribbling on its input, copy first.
- */
- params = copyObject(params);
-
- pstate = make_parsestate(NULL);
- pstate->p_sourcetext = queryString;
-
- i = 0;
- foreach(l, params)
- {
- Node *expr = lfirst(l);
- Oid expected_type_id = param_types[i];
- Oid given_type_id;
-
- expr = transformExpr(pstate, expr, EXPR_KIND_EXECUTE_PARAMETER);
-
- given_type_id = exprType(expr);
-
- expr = coerce_to_target_type(pstate, expr, given_type_id,
- expected_type_id, -1,
- COERCION_ASSIGNMENT,
- COERCE_IMPLICIT_CAST,
- -1);
-
- if (expr == NULL)
- ereport(ERROR,
- (errcode(ERRCODE_DATATYPE_MISMATCH),
- errmsg("parameter $%d of type %s cannot be coerced to the expected type %s",
- i + 1,
- format_type_be(given_type_id),
- format_type_be(expected_type_id)),
- errhint("You will need to rewrite or cast the expression.")));
-
- /* Take care of collations in the finished expression. */
- assign_expr_collations(pstate, expr);
-
- lfirst(l) = expr;
- i++;
- }
-
/* Prepare the expressions for execution */
exprstates = ExecPrepareExprList(params, estate);
@@ -655,8 +594,7 @@ ExplainExecuteQuery(ExecuteStmt *execstmt, IntoClause *into, ExplainState *es,
*/
estate = CreateExecutorState();
estate->es_param_list_info = params;
- paramLI = EvaluateParams(entry, execstmt->params,
- queryString, estate);
+ paramLI = EvaluateParams(entry, execstmt->params, estate);
}
/* Replan if needed, and acquire a transient refcount */
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 85d7a96406..943e1764bd 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -26,6 +26,7 @@
#include "access/sysattr.h"
#include "catalog/pg_type.h"
+#include "commands/prepare.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "nodes/nodeFuncs.h"
@@ -44,6 +45,7 @@
#include "parser/parse_target.h"
#include "parser/parsetree.h"
#include "rewrite/rewriteManip.h"
+#include "utils/builtins.h"
#include "utils/rel.h"
@@ -72,6 +74,7 @@ static List *transformUpdateTargetList(ParseState *pstate,
List *targetList);
static Query *transformDeclareCursorStmt(ParseState *pstate,
DeclareCursorStmt *stmt);
+static Query *transformExecuteStmt(ParseState *pstate, ExecuteStmt *stmt);
static Query *transformExplainStmt(ParseState *pstate,
ExplainStmt *stmt);
static Query *transformCreateTableAsStmt(ParseState *pstate,
@@ -312,6 +315,11 @@ transformStmt(ParseState *pstate, Node *parseTree)
(DeclareCursorStmt *) parseTree);
break;
+ case T_ExecuteStmt:
+ result = transformExecuteStmt(pstate,
+ (ExecuteStmt *) parseTree);
+ break;
+
case T_ExplainStmt:
result = transformExplainStmt(pstate,
(ExplainStmt *) parseTree);
@@ -2511,6 +2519,87 @@ transformDeclareCursorStmt(ParseState *pstate, DeclareCursorStmt *stmt)
return result;
}
+/*
+ * transformExexecuteStmt -
+ * transform an EXECUTE Statement
+ *
+ * This checks that the number of parameters passed to EXECUTE matches the
+ * prepared statement, and it transforms the passed parameter expressions.
+ */
+static Query *
+transformExecuteStmt(ParseState *pstate, ExecuteStmt *stmt)
+{
+ PreparedStatement *pstmt;
+ Oid *param_types;
+ int num_params;
+ int nparams;
+ List *params;
+ ListCell *l;
+ int i;
+ Query *result;
+
+ pstmt = FetchPreparedStatement(stmt->name, true);
+
+ param_types = pstmt->plansource->param_types;
+ num_params = pstmt->plansource->num_params;
+ nparams = list_length(stmt->params);
+
+ if (nparams != num_params)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("wrong number of parameters for prepared statement \"%s\"",
+ pstmt->stmt_name),
+ errdetail("Expected %d parameters but got %d.",
+ num_params, nparams)));
+
+ /*
+ * We have to run parse analysis for the expressions. Since the parser is
+ * not cool about scribbling on its input, copy first.
+ */
+ params = copyObject(stmt->params);
+ i = 0;
+ foreach(l, params)
+ {
+ Node *expr = lfirst(l);
+ Oid expected_type_id = param_types[i];
+ Oid given_type_id;
+
+ expr = transformExpr(pstate, expr, EXPR_KIND_EXECUTE_PARAMETER);
+
+ given_type_id = exprType(expr);
+
+ expr = coerce_to_target_type(pstate, expr, given_type_id,
+ expected_type_id, -1,
+ COERCION_ASSIGNMENT,
+ COERCE_IMPLICIT_CAST,
+ -1);
+
+ if (expr == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("parameter $%d of type %s cannot be coerced to the expected type %s",
+ i + 1,
+ format_type_be(given_type_id),
+ format_type_be(expected_type_id)),
+ errhint("You will need to rewrite or cast the expression."),
+ parser_errposition(pstate, exprLocation(lfirst(l)))));
+
+ /* Take care of collations in the finished expression. */
+ assign_expr_collations(pstate, expr);
+
+ lfirst(l) = expr;
+ i++;
+ }
+
+ stmt->params = params;
+
+ result = makeNode(Query);
+ result->commandType = CMD_UTILITY;
+ result->utilityStmt = (Node *) stmt;
+
+ return result;
+}
+
/*
* transformExplainStmt -
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index c6faa6619d..f2d00b0084 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -565,7 +565,7 @@ standard_ProcessUtility(PlannedStmt *pstmt,
case T_ExecuteStmt:
ExecuteQuery((ExecuteStmt *) parsetree, NULL,
- queryString, params,
+ params,
dest, completionTag);
break;
diff --git a/src/include/commands/prepare.h b/src/include/commands/prepare.h
index 2ce832419f..ff24797fce 100644
--- a/src/include/commands/prepare.h
+++ b/src/include/commands/prepare.h
@@ -38,7 +38,7 @@ typedef struct
extern void PrepareQuery(PrepareStmt *stmt, const char *queryString,
int stmt_location, int stmt_len);
extern void ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
- const char *queryString, ParamListInfo params,
+ ParamListInfo params,
DestReceiver *dest, char *completionTag);
extern void DeallocateQuery(DeallocateStmt *stmt);
extern void ExplainExecuteQuery(ExecuteStmt *execstmt, IntoClause *into,
diff --git a/src/test/regress/expected/prepare.out b/src/test/regress/expected/prepare.out
index 717732300d..a0fec13c6b 100644
--- a/src/test/regress/expected/prepare.out
+++ b/src/test/regress/expected/prepare.out
@@ -113,6 +113,8 @@ DETAIL: Expected 5 parameters but got 6.
-- wrong param types
EXECUTE q3(5::smallint, 10.5::float, false, 4::bigint, 'bytea');
ERROR: parameter $3 of type boolean cannot be coerced to the expected type double precision
+LINE 1: EXECUTE q3(5::smallint, 10.5::float, false, 4::bigint, 'byte...
+ ^
HINT: You will need to rewrite or cast the expression.
-- invalid type
PREPARE q4(nonexistenttype) AS SELECT $1;
@@ -185,3 +187,26 @@ SELECT name, statement, parameter_types FROM pg_prepared_statements
------+-----------+-----------------
(0 rows)
+-- check parameter handling
+CREATE TABLE t1 (a int);
+PREPARE p1 AS INSERT INTO t1 (a) VALUES ($1);
+CREATE FUNCTION f1(x int) RETURNS int
+LANGUAGE SQL
+AS $$
+EXECUTE p1($1);
+SELECT null::int;
+$$;
+SELECT f1(2);
+ f1
+----
+
+(1 row)
+
+SELECT * FROM t1;
+ a
+---
+ 2
+(1 row)
+
+DROP FUNCTION f1(int);
+DROP TABLE t1;
diff --git a/src/test/regress/sql/prepare.sql b/src/test/regress/sql/prepare.sql
index 985d0f05c9..6a16858482 100644
--- a/src/test/regress/sql/prepare.sql
+++ b/src/test/regress/sql/prepare.sql
@@ -78,3 +78,24 @@ CREATE TEMPORARY TABLE q5_prep_nodata AS EXECUTE q5(200, 'DTAAAA')
DEALLOCATE ALL;
SELECT name, statement, parameter_types FROM pg_prepared_statements
ORDER BY name;
+
+
+-- check parameter handling
+
+CREATE TABLE t1 (a int);
+
+PREPARE p1 AS INSERT INTO t1 (a) VALUES ($1);
+
+CREATE FUNCTION f1(x int) RETURNS int
+LANGUAGE SQL
+AS $$
+EXECUTE p1($1);
+SELECT null::int;
+$$;
+
+SELECT f1(2);
+
+SELECT * FROM t1;
+
+DROP FUNCTION f1(int);
+DROP TABLE t1;
--
2.23.0
Hello.
At Mon, 4 Nov 2019 08:53:18 +0100, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote in
On 2019-11-02 16:00, Tom Lane wrote:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt().Uhmm ... no actual patch attached?
Oops, here it is.
The patch just moves the first half of EvaluateParams that is
irrelevant to executor state to before portal parameters are set. I
looked with a suspect that extended protocol or SPI are affected but
AFAICS it doesn't seem to.
I dug into repository and found that transformExecuteStmt existed at
the time of implementing PREPARE-EXECUTE statements(28e82066a1) and
removed by the commit b9527e9840 which is related to
plan-invalidation.
git show -s --format=%B b9527e984092e838790b543b014c0c2720ea4f11
In service of this, rearrange utility-statement processing so that parse
analysis does not assume table schemas can't change before execution for
utility statements (necessary because we don't attempt to re-acquire locks
for utility statements when reusing a stored plan). This requires some
Isn't this related to the current structure?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
út 5. 11. 2019 v 11:28 odesílatel Kyotaro Horiguchi <horikyota.ntt@gmail.com>
napsal:
Hello.
At Mon, 4 Nov 2019 08:53:18 +0100, Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> wrote inOn 2019-11-02 16:00, Tom Lane wrote:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt().Uhmm ... no actual patch attached?
Oops, here it is.
The patch just moves the first half of EvaluateParams that is
irrelevant to executor state to before portal parameters are set. I
looked with a suspect that extended protocol or SPI are affected but
AFAICS it doesn't seem to.I dug into repository and found that transformExecuteStmt existed at
the time of implementing PREPARE-EXECUTE statements(28e82066a1) and
removed by the commit b9527e9840 which is related to
plan-invalidation.git show -s --format=%B b9527e984092e838790b543b014c0c2720ea4f11
In service of this, rearrange utility-statement processing so that parse
analysis does not assume table schemas can't change before execution for
utility statements (necessary because we don't attempt to re-acquirelocks
for utility statements when reusing a stored plan). This requires some
Isn't this related to the current structure?
I think so it should be ok, because the transformation is still in same
statement - if I understand well.
So visibility of system catalogue or access to plan cache should not be
changed.
Regards
Pavel
Show quoted text
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
po 4. 11. 2019 v 8:53 odesílatel Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> napsal:
On 2019-11-02 16:00, Tom Lane wrote:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
This patch moves the parse analysis component of ExecuteQuery() and
EvaluateParams() into a new transformExecuteStmt() that is called from
transformStmt().Uhmm ... no actual patch attached?
Oops, here it is.
I checked this patch, and I think so it's correct and wanted. It introduce
transform stage for EXECUTE command, and move there the argument
transformation.
This has sensible change - the code is much more correct now.
The patching, compilation was without any problems, make check-world too.
I was little bit confused about regress tests - the patch did some code
refactoring and I expect so main target is same behave before and after
patching. But the regress tests shows new feature that is just side effect
(nice) of patch. More, the example is little bit strange - nobody will use
prepared statements and execution in SQL function. It should be better
commented.
I'll mark this patch as ready for commiters.
Regards
Pavel
Show quoted text
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-11-08 08:13, Pavel Stehule wrote:
I dug into repository and found that transformExecuteStmt existed at
the time of implementing PREPARE-EXECUTE statements(28e82066a1) and
removed by the commit b9527e9840 which is related to
plan-invalidation.git show -s --format=%B b9527e984092e838790b543b014c0c2720ea4f11
In service of this, rearrange utility-statement processing so
that parse
analysis does not assume table schemas can't change before
execution for
utility statements (necessary because we don't attempt to
re-acquire locks
for utility statements when reusing a stored plan). This
requires some
Isn't this related to the current structure?
I think so it should be ok, because the transformation is still in same
statement - if I understand well.So visibility of system catalogue or access to plan cache should not be
changed.
I think what that patch was addressing is, if you use a protocol-level
prepare+execute with commands like CREATE INDEX, CREATE VIEW, or COPY
and you change the table schema between the prepare and execute, things
would break, for the reasons explained in the commit message. So any
parse analysis in utility statements that accesses table schemas needs
to be done in the execute phase, not in the prepare phase, as one might
think.
Parse analysis of EXECUTE does not access any tables, so if I understood
this correctly, this concern doesn't apply here.
Interestingly, the above commit also removed the prepare-time
transformation of ExplainStmt, but it was later put back and now has the
comment "We used to postpone that until execution, but it's really
necessary to do it during the normal parse analysis phase to ensure that
side effects of parser hooks happen at the expected time." So there
appears to be a generally uneasy situation still about how to do this
correctly.
Perhaps something could be done about the issue "because we don't
attempt to re-acquire locks for utility statements when reusing a stored
plan"?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
pá 8. 11. 2019 v 8:54 odesílatel Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> napsal:
On 2019-11-08 08:13, Pavel Stehule wrote:
I dug into repository and found that transformExecuteStmt existed at
the time of implementing PREPARE-EXECUTE statements(28e82066a1) and
removed by the commit b9527e9840 which is related to
plan-invalidation.git show -s --format=%B b9527e984092e838790b543b014c0c2720ea4f11
In service of this, rearrange utility-statement processing so
that parse
analysis does not assume table schemas can't change before
execution for
utility statements (necessary because we don't attempt to
re-acquire locks
for utility statements when reusing a stored plan). This
requires some
Isn't this related to the current structure?
I think so it should be ok, because the transformation is still in same
statement - if I understand well.So visibility of system catalogue or access to plan cache should not be
changed.I think what that patch was addressing is, if you use a protocol-level
prepare+execute with commands like CREATE INDEX, CREATE VIEW, or COPY
and you change the table schema between the prepare and execute, things
would break, for the reasons explained in the commit message. So any
parse analysis in utility statements that accesses table schemas needs
to be done in the execute phase, not in the prepare phase, as one might
think.Parse analysis of EXECUTE does not access any tables, so if I understood
this correctly, this concern doesn't apply here.
it should not be true - the subquery can be a expression.
Minimally on SQL level is not possible do prepare on execute. So execute
should be evaluate as one step.
Show quoted text
Interestingly, the above commit also removed the prepare-time
transformation of ExplainStmt, but it was later put back and now has the
comment "We used to postpone that until execution, but it's really
necessary to do it during the normal parse analysis phase to ensure that
side effects of parser hooks happen at the expected time." So there
appears to be a generally uneasy situation still about how to do this
correctly.Perhaps something could be done about the issue "because we don't
attempt to re-acquire locks for utility statements when reusing a stored
plan"?--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-11-08 09:03, Pavel Stehule wrote:
Parse analysis of EXECUTE does not access any tables, so if I
understood
this correctly, this concern doesn't apply here.it should not be true - the subquery can be a expression.
Arguments of EXECUTE cannot be subqueries.
Minimally on SQL level is not possible do prepare on execute. So execute
should be evaluate as one step.
Well, that's kind of the question that is being discussed in this thread.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
pá 8. 11. 2019 v 13:34 odesílatel Peter Eisentraut <
peter.eisentraut@2ndquadrant.com> napsal:
On 2019-11-08 09:03, Pavel Stehule wrote:
Parse analysis of EXECUTE does not access any tables, so if I
understood
this correctly, this concern doesn't apply here.it should not be true - the subquery can be a expression.
Arguments of EXECUTE cannot be subqueries.
ok
Minimally on SQL level is not possible do prepare on execute. So execute
should be evaluate as one step.Well, that's kind of the question that is being discussed in this thread.
I say it not cleanly - I think so this change should be safe, because
parsing, transforming, and execution must be in one statement.
Regards
Pavel
Show quoted text
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
On 2019-11-08 09:03, Pavel Stehule wrote:
Minimally on SQL level is not possible do prepare on execute. So execute
should be evaluate as one step.
Well, that's kind of the question that is being discussed in this thread.
Yeah. Having now taken a quick look at this patch, it makes me pretty
queasy. In particular, it doesn't appear to add any support for
invalidation of cached EXECUTE commands when their parameter expressions
change. You dismissed that as irrelevant because no table schemas would
be involved, but there's also the possibility of replacements of user
defined functions. I'm not sure how easy it is to create a situation
where an EXECUTE statement is in plancache, but it's probably possible
(maybe using some other PL than plpgsql). In that case, we really would
need the EXECUTE's transformed expressions to get invalidated if the
user drops or replaces a function they use.
In view of the ALTER TABLE bugs I'm struggling with over in [1]/messages/by-id/10365.1558909428@sss.pgh.pa.us, I feel
like this patch is probably going in the wrong direction. We should
generally be striving to do all transformation of utility commands as
late as possible. As long as a plancached utility statement contains
nothing beyond raw-parser output, it never needs invalidation.
You pointed to an old comment of mine about EXPLAIN that seems to argue
in the other direction, but digging in the commit log, I see that it
came from commit 08f8d478, whose log entry is perhaps more informative
than the comment:
Do parse analysis of an EXPLAIN's contained statement during the normal
parse analysis phase, rather than at execution time. This makes parameter
handling work the same as it does in ordinary plannable queries, and in
particular fixes the incompatibility that Pavel pointed out with plpgsql's
new handling of variable references. plancache.c gets a little bit
grottier, but the alternatives seem worse.
So what this really is all about is still the same old issue of how we
handle external parameter references in utility statements. Maybe we
ought to focus on a redesign addressing that specific problem, rather
than nibbling around the edges. It seems like the core of the issue
is that we have mechanisms for PLs to capture parameter references
during parse analysis, and those hooks aren't managed in a way that
lets them be invoked if we do parse analysis during utility statement
execution. But we *need* to be able to do that. ALTER TABLE already
does do that, yet we need to postpone its analysis to even later than
it's doing it now.
Another issue in all this is that for many utility statements, you
don't actually want injections of PL parameter references, for instance
it'd make little sense to allow "alter table ... add check (f1 > p1)"
if p1 is a local variable in the function doing the ALTER. It's
probably time to have some explicit recognition and management of such
cases, rather than just dodging them by not invoking the hooks.
tl;dr: I think that we need to embrace parse analysis during utility
statement execution as a fully supported thing, not a stepchild.
Trying to make it go away is the wrong approach.
regards, tom lane
After digesting the discussion, let's reshuffle this a bit.
I have committed the change that adds the error location in one place.
That worked independently.
Attached is a new patch that refactors things a bit to pass the
ParseState into functions such as PrepareQuery() and ExecuteQuery()
instead of passing the query string and query environment as a separate
arguments. We had already done that for most utility commands; there
were just some left that happened to be involved in the current thread's
discussion anyway.
That's a nice cosmetic improvement in any case, but I think that it
would also help with the issue of passing parameters into some utility
commands later on. I will look into that some other time.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v2-0001-Make-better-use-of-ParseState-in-ProcessUtility.patchtext/plain; charset=UTF-8; name=v2-0001-Make-better-use-of-ParseState-in-ProcessUtility.patch; x-mac-creator=0; x-mac-type=0Download
From d4580b9750523f4cfde8c90b8304110f66967ebe Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter@eisentraut.org>
Date: Mon, 25 Nov 2019 09:15:56 +0100
Subject: [PATCH v2] Make better use of ParseState in ProcessUtility
Pass ParseState into the functions called from
standard_ProcessUtility() instead passing the query string and query
environment separately. No functionality change, but it makes the
notation consistent. We had already started moving things into
that direction piece by piece, but this completes it.
---
src/backend/commands/createas.c | 6 ++--
src/backend/commands/explain.c | 7 ++---
src/backend/commands/portalcmds.c | 9 +++---
src/backend/commands/prepare.c | 52 ++++++++++++-------------------
src/backend/tcop/utility.c | 20 ++++++------
src/include/commands/createas.h | 2 +-
src/include/commands/explain.h | 4 +--
src/include/commands/portalcmds.h | 5 +--
src/include/commands/prepare.h | 7 +++--
9 files changed, 51 insertions(+), 61 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index 2bf7083719..d413a7c80d 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -223,7 +223,7 @@ create_ctas_nodata(List *tlist, IntoClause *into)
* ExecCreateTableAs -- execute a CREATE TABLE AS command
*/
ObjectAddress
-ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
+ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
ParamListInfo params, QueryEnvironment *queryEnv,
char *completionTag)
{
@@ -270,7 +270,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
ExecuteStmt *estmt = castNode(ExecuteStmt, query->utilityStmt);
Assert(!is_matview); /* excluded by syntax */
- ExecuteQuery(estmt, into, queryString, params, dest, completionTag);
+ ExecuteQuery(pstate, estmt, into, params, dest, completionTag);
/* get object address that intorel_startup saved for us */
address = ((DR_intorel *) dest)->reladdr;
@@ -342,7 +342,7 @@ ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
UpdateActiveSnapshotCommandId();
/* Create a QueryDesc, redirecting output to our tuple receiver */
- queryDesc = CreateQueryDesc(plan, queryString,
+ queryDesc = CreateQueryDesc(plan, pstate->p_sourcetext,
GetActiveSnapshot(), InvalidSnapshot,
dest, params, queryEnv, 0);
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 62fb3434a3..7d174ec739 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -139,9 +139,8 @@ static void escape_yaml(StringInfo buf, const char *str);
* execute an EXPLAIN command
*/
void
-ExplainQuery(ParseState *pstate, ExplainStmt *stmt, const char *queryString,
- ParamListInfo params, QueryEnvironment *queryEnv,
- DestReceiver *dest)
+ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
+ ParamListInfo params, DestReceiver *dest)
{
ExplainState *es = NewExplainState();
TupOutputState *tstate;
@@ -254,7 +253,7 @@ ExplainQuery(ParseState *pstate, ExplainStmt *stmt, const char *queryString,
{
ExplainOneQuery(lfirst_node(Query, l),
CURSOR_OPT_PARALLEL_OK, NULL, es,
- queryString, params, queryEnv);
+ pstate->p_sourcetext, params, pstate->p_queryEnv);
/* Separate plans with an appropriate separator */
if (lnext(rewritten, l) != NULL)
diff --git a/src/backend/commands/portalcmds.c b/src/backend/commands/portalcmds.c
index 83f9959d54..e100cf379c 100644
--- a/src/backend/commands/portalcmds.c
+++ b/src/backend/commands/portalcmds.c
@@ -39,14 +39,15 @@
* Execute SQL DECLARE CURSOR command.
*/
void
-PerformCursorOpen(DeclareCursorStmt *cstmt, ParamListInfo params,
- const char *queryString, bool isTopLevel)
+PerformCursorOpen(ParseState *pstate, DeclareCursorStmt *cstmt, ParamListInfo params,
+ bool isTopLevel)
{
Query *query = castNode(Query, cstmt->query);
List *rewritten;
PlannedStmt *plan;
Portal portal;
MemoryContext oldContext;
+ char *queryString;
/*
* Disallow empty-string cursor name (conflicts with protocol-level
@@ -92,7 +93,7 @@ PerformCursorOpen(DeclareCursorStmt *cstmt, ParamListInfo params,
plan = pg_plan_query(query, cstmt->options, params);
/*
- * Create a portal and copy the plan and queryString into its memory.
+ * Create a portal and copy the plan and query string into its memory.
*/
portal = CreatePortal(cstmt->portalname, false, false);
@@ -100,7 +101,7 @@ PerformCursorOpen(DeclareCursorStmt *cstmt, ParamListInfo params,
plan = copyObject(plan);
- queryString = pstrdup(queryString);
+ queryString = pstrdup(pstate->p_sourcetext);
PortalDefineQuery(portal,
NULL,
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 47bae957dc..604dd3fac6 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -46,15 +46,16 @@
static HTAB *prepared_queries = NULL;
static void InitQueryHashTable(void);
-static ParamListInfo EvaluateParams(PreparedStatement *pstmt, List *params,
- const char *queryString, EState *estate);
+static ParamListInfo EvaluateParams(ParseState *pstate,
+ PreparedStatement *pstmt, List *params,
+ EState *estate);
static Datum build_regtype_array(Oid *param_types, int num_params);
/*
* Implements the 'PREPARE' utility statement.
*/
void
-PrepareQuery(PrepareStmt *stmt, const char *queryString,
+PrepareQuery(ParseState *pstate, PrepareStmt *stmt,
int stmt_location, int stmt_len)
{
RawStmt *rawstmt;
@@ -90,7 +91,7 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString,
* Create the CachedPlanSource before we do parse analysis, since it needs
* to see the unmodified raw parse tree.
*/
- plansource = CreateCachedPlan(rawstmt, queryString,
+ plansource = CreateCachedPlan(rawstmt, pstate->p_sourcetext,
CreateCommandTag(stmt->query));
/* Transform list of TypeNames to array of type OIDs */
@@ -98,16 +99,8 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString,
if (nargs)
{
- ParseState *pstate;
ListCell *l;
- /*
- * typenameTypeId wants a ParseState to carry the source query string.
- * Is it worth refactoring its API to avoid this?
- */
- pstate = make_parsestate(NULL);
- pstate->p_sourcetext = queryString;
-
argtypes = (Oid *) palloc(nargs * sizeof(Oid));
i = 0;
@@ -125,7 +118,7 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString,
* passed in from above us will not be visible to it), allowing
* information about unknown parameters to be deduced from context.
*/
- query = parse_analyze_varparams(rawstmt, queryString,
+ query = parse_analyze_varparams(rawstmt, pstate->p_sourcetext,
&argtypes, &nargs);
/*
@@ -189,16 +182,11 @@ PrepareQuery(PrepareStmt *stmt, const char *queryString,
* indicated by passing a non-null intoClause. The DestReceiver is already
* set up correctly for CREATE TABLE AS, but we still have to make a few
* other adjustments here.
- *
- * Note: this is one of very few places in the code that needs to deal with
- * two query strings at once. The passed-in queryString is that of the
- * EXECUTE, which we might need for error reporting while processing the
- * parameter expressions. The query_string that we copy from the plan
- * source is that of the original PREPARE.
*/
void
-ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
- const char *queryString, ParamListInfo params,
+ExecuteQuery(ParseState *pstate,
+ ExecuteStmt *stmt, IntoClause *intoClause,
+ ParamListInfo params,
DestReceiver *dest, char *completionTag)
{
PreparedStatement *entry;
@@ -229,8 +217,7 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
*/
estate = CreateExecutorState();
estate->es_param_list_info = params;
- paramLI = EvaluateParams(entry, stmt->params,
- queryString, estate);
+ paramLI = EvaluateParams(pstate, entry, stmt->params, estate);
}
/* Create a new portal to run the query in */
@@ -314,9 +301,9 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
/*
* EvaluateParams: evaluate a list of parameters.
*
+ * pstate: parse state
* pstmt: statement we are getting parameters for.
* params: list of given parameter expressions (raw parser output!)
- * queryString: source text for error messages.
* estate: executor state to use.
*
* Returns a filled-in ParamListInfo -- this can later be passed to
@@ -324,13 +311,12 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
* during query execution.
*/
static ParamListInfo
-EvaluateParams(PreparedStatement *pstmt, List *params,
- const char *queryString, EState *estate)
+EvaluateParams(ParseState *pstate, PreparedStatement *pstmt, List *params,
+ EState *estate)
{
Oid *param_types = pstmt->plansource->param_types;
int num_params = pstmt->plansource->num_params;
int nparams = list_length(params);
- ParseState *pstate;
ParamListInfo paramLI;
List *exprstates;
ListCell *l;
@@ -354,9 +340,6 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
*/
params = copyObject(params);
- pstate = make_parsestate(NULL);
- pstate->p_sourcetext = queryString;
-
i = 0;
foreach(l, params)
{
@@ -648,6 +631,11 @@ ExplainExecuteQuery(ExecuteStmt *execstmt, IntoClause *into, ExplainState *es,
/* Evaluate parameters, if any */
if (entry->plansource->num_params)
{
+ ParseState *pstate;
+
+ pstate = make_parsestate(NULL);
+ pstate->p_sourcetext = queryString;
+
/*
* Need an EState to evaluate parameters; must not delete it till end
* of query, in case parameters are pass-by-reference. Note that the
@@ -656,8 +644,8 @@ ExplainExecuteQuery(ExecuteStmt *execstmt, IntoClause *into, ExplainState *es,
*/
estate = CreateExecutorState();
estate->es_param_list_info = params;
- paramLI = EvaluateParams(entry, execstmt->params,
- queryString, estate);
+
+ paramLI = EvaluateParams(pstate, entry, execstmt->params, estate);
}
/* Replan if needed, and acquire a transient refcount */
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index 3a03ca7e2f..0fb8c694bd 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -396,6 +396,7 @@ standard_ProcessUtility(PlannedStmt *pstmt,
pstate = make_parsestate(NULL);
pstate->p_sourcetext = queryString;
+ pstate->p_queryEnv = queryEnv;
switch (nodeTag(parsetree))
{
@@ -500,8 +501,8 @@ standard_ProcessUtility(PlannedStmt *pstmt,
* Portal (cursor) manipulation
*/
case T_DeclareCursorStmt:
- PerformCursorOpen((DeclareCursorStmt *) parsetree, params,
- queryString, isTopLevel);
+ PerformCursorOpen(pstate, (DeclareCursorStmt *) parsetree, params,
+ isTopLevel);
break;
case T_ClosePortalStmt:
@@ -558,13 +559,14 @@ standard_ProcessUtility(PlannedStmt *pstmt,
case T_PrepareStmt:
CheckRestrictedOperation("PREPARE");
- PrepareQuery((PrepareStmt *) parsetree, queryString,
+ PrepareQuery(pstate, (PrepareStmt *) parsetree,
pstmt->stmt_location, pstmt->stmt_len);
break;
case T_ExecuteStmt:
- ExecuteQuery((ExecuteStmt *) parsetree, NULL,
- queryString, params,
+ ExecuteQuery(pstate,
+ (ExecuteStmt *) parsetree, NULL,
+ params,
dest, completionTag);
break;
@@ -667,8 +669,7 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
case T_ExplainStmt:
- ExplainQuery(pstate, (ExplainStmt *) parsetree, queryString, params,
- queryEnv, dest);
+ ExplainQuery(pstate, (ExplainStmt *) parsetree, params, dest);
break;
case T_AlterSystemStmt:
@@ -1490,9 +1491,8 @@ ProcessUtilitySlow(ParseState *pstate,
break;
case T_CreateTableAsStmt:
- address = ExecCreateTableAs((CreateTableAsStmt *) parsetree,
- queryString, params, queryEnv,
- completionTag);
+ address = ExecCreateTableAs(pstate, (CreateTableAsStmt *) parsetree,
+ params, queryEnv, completionTag);
break;
case T_RefreshMatViewStmt:
diff --git a/src/include/commands/createas.h b/src/include/commands/createas.h
index abd0525290..a5467bcba4 100644
--- a/src/include/commands/createas.h
+++ b/src/include/commands/createas.h
@@ -21,7 +21,7 @@
#include "utils/queryenvironment.h"
-extern ObjectAddress ExecCreateTableAs(CreateTableAsStmt *stmt, const char *queryString,
+extern ObjectAddress ExecCreateTableAs(ParseState *pstate, CreateTableAsStmt *stmt,
ParamListInfo params, QueryEnvironment *queryEnv, char *completionTag);
extern int GetIntoRelEFlags(IntoClause *intoClause);
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index 8639891c16..98d5b46337 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -63,8 +63,8 @@ typedef const char *(*explain_get_index_name_hook_type) (Oid indexId);
extern PGDLLIMPORT explain_get_index_name_hook_type explain_get_index_name_hook;
-extern void ExplainQuery(ParseState *pstate, ExplainStmt *stmt, const char *queryString,
- ParamListInfo params, QueryEnvironment *queryEnv, DestReceiver *dest);
+extern void ExplainQuery(ParseState *pstate, ExplainStmt *stmt,
+ ParamListInfo params, DestReceiver *dest);
extern ExplainState *NewExplainState(void);
diff --git a/src/include/commands/portalcmds.h b/src/include/commands/portalcmds.h
index 73220981ee..640e663ed7 100644
--- a/src/include/commands/portalcmds.h
+++ b/src/include/commands/portalcmds.h
@@ -15,11 +15,12 @@
#define PORTALCMDS_H
#include "nodes/parsenodes.h"
+#include "parser/parse_node.h"
#include "utils/portal.h"
-extern void PerformCursorOpen(DeclareCursorStmt *cstmt, ParamListInfo params,
- const char *queryString, bool isTopLevel);
+extern void PerformCursorOpen(ParseState *pstate, DeclareCursorStmt *cstmt, ParamListInfo params,
+ bool isTopLevel);
extern void PerformPortalFetch(FetchStmt *stmt, DestReceiver *dest,
char *completionTag);
diff --git a/src/include/commands/prepare.h b/src/include/commands/prepare.h
index 2ce832419f..c828f62473 100644
--- a/src/include/commands/prepare.h
+++ b/src/include/commands/prepare.h
@@ -35,10 +35,11 @@ typedef struct
/* Utility statements PREPARE, EXECUTE, DEALLOCATE, EXPLAIN EXECUTE */
-extern void PrepareQuery(PrepareStmt *stmt, const char *queryString,
+extern void PrepareQuery(ParseState *pstate, PrepareStmt *stmt,
int stmt_location, int stmt_len);
-extern void ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
- const char *queryString, ParamListInfo params,
+extern void ExecuteQuery(ParseState *pstate,
+ ExecuteStmt *stmt, IntoClause *intoClause,
+ ParamListInfo params,
DestReceiver *dest, char *completionTag);
extern void DeallocateQuery(DeallocateStmt *stmt);
extern void ExplainExecuteQuery(ExecuteStmt *execstmt, IntoClause *into,
--
2.24.0
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
This patch replaced query string by parse state on few places. It increase code consistency.
The new status of this patch is: Ready for Committer
On 2020-01-02 14:26, Pavel Stehule wrote:
The following review has been posted through the commitfest application:
make installcheck-world: tested, passed
Implements feature: not tested
Spec compliant: not tested
Documentation: not testedThis patch replaced query string by parse state on few places. It increase code consistency.
The new status of this patch is: Ready for Committer
committed, thanks
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services