pgbench: extend variable usage in scripts

Started by Yugo Nagata5 months ago5 messages
#1lakshmi
lakshmigcdac@gmail.com
In reply to: Michael Paquier (#5)
1 attachment(s)
Re: pgbench: extend variable usage in scripts

Hi,

While testing
patch (0001-pgbench-Allow-variables-to-be-used-as-an-SQL-literal.patch), I
reproduced the CI failure reported by Michael (-Werror=switch for
PQUOTE_SHELL_ARG in replaceVariable()).

Attached is a small follow-up patch that handles
-The missing enum case,
-Removes an unused variable
-Ensures quoted_value is always initialized.
This builds cleanly for me.

Regards,
Lakshmi

On Fri, Jan 9, 2026 at 2:41 PM Michael Paquier <michael@paquier.xyz> wrote:

Show quoted text

On Thu, Sep 18, 2025 at 03:33:30PM +0900, Yugo Nagata wrote:

Any thought?

Note that the CI is complaining a bit. Relevant extract:
[22:51:36.908] pgbench.c: In function ‘replaceVariable’:
[22:51:36.908] pgbench.c:1990:9: error: enumeration value
‘PQUOTE_SHELL_ARG’ not handled in switch [-Werror=switch]
[22:51:36.908] 1990 | switch (quote)
--
Michael

Attachments:

pgbench-ci-fix.patchtext/x-patch; charset=US-ASCII; name=pgbench-ci-fix.patchDownload
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index a10e852f77d..102e64902f9 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1953,14 +1953,17 @@ static char *
 replaceVariable(char **sql, char *param, int len, char *value, PsqlScanQuoteType quote, PGconn *conn)
 {
 	int			valueln;
-	char		quotechar;
-	char	   *quoted_value;
+	char	   *quoted_value	=	NULL;
 
 	switch (quote)
 	{
 		case PQUOTE_PLAIN:
 			quoted_value = pg_strdup(value);
 			break;
+		case PQUOTE_SHELL_ARG:
+			quoted_value = pg_strdup(value);
+			break;
+
 		case PQUOTE_SQL_LITERAL:
 		case PQUOTE_SQL_IDENT:
 			{
@@ -1995,7 +1998,11 @@ replaceVariable(char **sql, char *param, int len, char *value, PsqlScanQuoteType
 				break;
 			}
 	}
-
+	if (quoted_value == NULL)
+	{
+		pg_log_error("unhandled quoting case");
+		exit(1);
+	}
 	valueln = strlen(quoted_value);
 
 	if (valueln > len)
#2Yugo Nagata
nagata@sraoss.co.jp
2 attachment(s)

Hi,

I would like to propose patches to extend variable usage in pgbench scripts.

* 0001: Allow variables to be used as an SQL literal or identifier

Currently, variables in pgbench scripts are expanded always without
quotes, so they cannot be used as SQL literals or identifiers.
This patch allows the use of :'var' and :"var" in addition to :var

However, they can used only in SQL commands, not in the arguments of
meta-commands, since exprscan.l (the lexical scanner for pgbench backslash
commands) cannot currently handle quoted values.

Also, we have to use \aset, \gset or -D option to assign a string to a varialbe,
since pgbench's \set command cannot recognize text values in its arguments.

* 0002: Add syntax for variable exisitence check

Currently, pgbench does not support :{?var} syntax to check whether the variable
is defined or not. This patch adds support for this syntax in meta-command arguments.
This is useful for checking if \aset set the result to the variable or not.

However, it cannot be used in SQL statements for now.

Some of the current limitations described above might be relaxed in the future, but
for now, I would like to ask for initial feedback on this proposal.

Regards,
Yugo Nagata

--
Yugo Nagata <nagata@sraoss.co.jp>

Attachments:

0002-pgbench-Add-syntax-for-variable-exisitence-check.patchtext/x-diff; name=0002-pgbench-Add-syntax-for-variable-exisitence-check.patchDownload
From a3f262cb96733cc041a7df1a43fe0f81c71f1474 Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Wed, 13 Aug 2025 19:16:06 +0900
Subject: [PATCH 2/2] pgbench: Add syntax for variable exisitence check

Now, :{?var} can be used in meta-command arguments. It can not
be used in SQL statements.
---
 src/bin/pgbench/exprscan.l |  4 +++-
 src/bin/pgbench/pgbench.c  | 27 +++++++++++++++++++++++++--
 2 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/src/bin/pgbench/exprscan.l b/src/bin/pgbench/exprscan.l
index 5747af38cb2..fb14f5276e7 100644
--- a/src/bin/pgbench/exprscan.l
+++ b/src/bin/pgbench/exprscan.l
@@ -182,12 +182,14 @@ notnull			[Nn][Oo][Tt][Nn][Uu][Ll][Ll]
 {else}			{ return ELSE_KW; }
 {end}			{ return END_KW; }
 
-:{alnum}+		{
+:{alnum}+		|
+:\{\?{alnum}+\}	{
 					yylval->str = pg_strdup(yytext + 1);
 					return VARIABLE;
 				}
 
 {null}			{ return NULL_CONST; }
+
 {true}			{
 					yylval->bval = true;
 					return BOOLEAN_CONST;
diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 14d1261ff8d..ae0b9ccaf83 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1920,7 +1920,6 @@ parseVariable(const char *sql, int *eaten, PsqlScanQuoteType *quote)
 	int			start;
 	int			len;
 
-
 	if (sql[i] == '\'')
 	{
 		i++;
@@ -2958,12 +2957,36 @@ evaluateExpr(CState *st, PgBenchExpr *expr, PgBenchValue *retval)
 		case ENODE_VARIABLE:
 			{
 				Variable   *var;
+				char	   *varname = expr->u.variable.varname;
+				bool		is_test_var = false;
+
+				if (varname[0] == '{' && varname[1] == '?')
+				{
+					int len;
+
+					is_test_var = true;
+					varname = pg_strdup(varname + 2);
+					len = strlen(varname);
+					varname[len - 1] = '\0';
+				}
 
-				if ((var = lookupVariable(&st->variables, expr->u.variable.varname)) == NULL)
+				if ((var = lookupVariable(&st->variables, varname)) == NULL)
 				{
 					pg_log_error("undefined variable \"%s\"", expr->u.variable.varname);
+					if (is_test_var)
+					{
+						free(varname);
+						setBoolValue(retval, false);
+						return true;
+					}
 					return false;
 				}
+				else if (is_test_var)
+				{
+					setBoolValue(retval, true);
+					free(varname);
+					return true;
+				}
 
 				if (!makeVariableValue(var))
 					return false;
-- 
2.43.0

0001-pgbench-Allow-variables-to-be-used-as-an-SQL-literal.patchtext/x-diff; name=0001-pgbench-Allow-variables-to-be-used-as-an-SQL-literal.patchDownload
From bae717f8b2710a6e29f0acf5b52de5f5501b8ccc Mon Sep 17 00:00:00 2001
From: Yugo Nagata <nagata@sraoss.co.jp>
Date: Tue, 12 Aug 2025 01:08:24 +0900
Subject: [PATCH 1/2] pgbench: Allow variables to be used as an SQL literal or
 identifier

Now, the form :'var' and :"var" can be used in addition to :var,
but only in the SQL command. They do not work in arguments of
meta-commands for now.
---
 src/bin/pgbench/pgbench.c | 116 +++++++++++++++++++++++++++++++++-----
 1 file changed, 103 insertions(+), 13 deletions(-)

diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c
index 125f3c7bbbe..14d1261ff8d 100644
--- a/src/bin/pgbench/pgbench.c
+++ b/src/bin/pgbench/pgbench.c
@@ -1913,10 +1913,32 @@ putVariableInt(Variables *variables, const char *context, char *name,
  * Otherwise, return NULL.
  */
 static char *
-parseVariable(const char *sql, int *eaten)
+parseVariable(const char *sql, int *eaten, PsqlScanQuoteType *quote)
 {
 	int			i = 1;			/* starting at 1 skips the colon */
 	char	   *name;
+	int			start;
+	int			len;
+
+
+	if (sql[i] == '\'')
+	{
+		i++;
+		*quote = PQUOTE_SQL_LITERAL;
+		start = 2;
+	}
+	else if (sql[i] == '"')
+	{
+		i++;
+		*quote = PQUOTE_SQL_IDENT;
+		start = 2;
+	}
+	else
+	{
+		*quote = PQUOTE_PLAIN;
+		start = 1;
+	}
+
 
 	/* keep this logic in sync with valid_variable_name() */
 	if (IS_HIGHBIT_SET(sql[i]) ||
@@ -1931,18 +1953,82 @@ parseVariable(const char *sql, int *eaten)
 				  "_0123456789", sql[i]) != NULL)
 		i++;
 
-	name = pg_malloc(i);
-	memcpy(name, &sql[1], i - 1);
-	name[i - 1] = '\0';
+	if (*quote == PQUOTE_SQL_LITERAL)
+	{
+		if (sql[i] != '\'')
+			return NULL;
+		i++;
+		len = i - 3;
+	}
+	else if (*quote == PQUOTE_SQL_IDENT)
+	{
+		if (sql[i] != '"')
+			return NULL;
+		i++;
+		len = i - 3;
+	}
+	else
+	{
+		Assert(*quote == PQUOTE_PLAIN);
+		len = i - 1;
+	}
+
+	name = pg_malloc(len + 1);
+	memcpy(name, &sql[start], len);
+	name[len] = '\0';
 
 	*eaten = i;
 	return name;
 }
 
 static char *
-replaceVariable(char **sql, char *param, int len, char *value)
+replaceVariable(char **sql, char *param, int len, char *value, PsqlScanQuoteType quote, PGconn *conn)
 {
-	int			valueln = strlen(value);
+	int			valueln;
+	char		quotechar;
+	char	   *quoted_value;
+
+	switch (quote)
+	{
+		case PQUOTE_PLAIN:
+			quoted_value = pg_strdup(value);
+			break;
+		case PQUOTE_SQL_LITERAL:
+		case PQUOTE_SQL_IDENT:
+			{
+				/*
+				 * For these cases, we use libpq's quoting functions, which
+				 * assume the string is in the connection's client encoding.
+				 */
+				char	   *escaped_value;
+
+				Assert(conn);
+
+				if (quote == PQUOTE_SQL_LITERAL)
+					escaped_value =
+						PQescapeLiteral(conn, value, strlen(value));
+				else
+					escaped_value =
+						PQescapeIdentifier(conn, value, strlen(value));
+
+				if (escaped_value == NULL)
+				{
+					pg_log_error("escape failed: %s", PQerrorMessage(conn));
+					exit(1);
+				}
+
+				/*
+				 * Rather than complicate the lexer's API with a notion of
+				 * which free() routine to use, just pay the price of an extra
+				 * strdup().
+				 */
+				quoted_value = pg_strdup(escaped_value);
+				PQfreemem(escaped_value);
+				break;
+			}
+	}
+
+	valueln = strlen(quoted_value);
 
 	if (valueln > len)
 	{
@@ -1954,13 +2040,15 @@ replaceVariable(char **sql, char *param, int len, char *value)
 
 	if (valueln != len)
 		memmove(param + valueln, param + len, strlen(param + len) + 1);
-	memcpy(param, value, valueln);
+	memcpy(param, quoted_value, valueln);
+
+	pfree(quoted_value);
 
 	return param + valueln;
 }
 
 static char *
-assignVariables(Variables *variables, char *sql)
+assignVariables(Variables *variables, char *sql, PGconn *conn)
 {
 	char	   *p,
 			   *name,
@@ -1970,8 +2058,9 @@ assignVariables(Variables *variables, char *sql)
 	while ((p = strchr(p, ':')) != NULL)
 	{
 		int			eaten;
+		PsqlScanQuoteType quote;
 
-		name = parseVariable(p, &eaten);
+		name = parseVariable(p, &eaten, &quote);
 		if (name == NULL)
 		{
 			while (*p == ':')
@@ -1989,7 +2078,7 @@ assignVariables(Variables *variables, char *sql)
 			continue;
 		}
 
-		p = replaceVariable(&sql, p, eaten, val);
+		p = replaceVariable(&sql, p, eaten, val, quote, conn);
 	}
 
 	return sql;
@@ -3183,7 +3272,7 @@ sendCommand(CState *st, Command *command)
 		char	   *sql;
 
 		sql = pg_strdup(command->argv[0]);
-		sql = assignVariables(&st->variables, sql);
+		sql = assignVariables(&st->variables, sql, st->con);
 
 		pg_log_debug("client %d sending %s", st->id, sql);
 		r = PQsendQuery(st->con, sql);
@@ -5480,8 +5569,9 @@ parseQuery(Command *cmd)
 		char		var[13];
 		char	   *name;
 		int			eaten;
+		PsqlScanQuoteType quote;
 
-		name = parseVariable(p, &eaten);
+		name = parseVariable(p, &eaten, &quote);
 		if (name == NULL)
 		{
 			while (*p == ':')
@@ -5504,7 +5594,7 @@ parseQuery(Command *cmd)
 		}
 
 		sprintf(var, "$%d", cmd->argc);
-		p = replaceVariable(&sql, p, eaten, var);
+		p = replaceVariable(&sql, p, eaten, var, PQUOTE_PLAIN, NULL);
 
 		cmd->argv[cmd->argc] = name;
 		cmd->argc++;
-- 
2.43.0

#3Pavel Stehule
pavel.stehule@gmail.com
In reply to: Yugo Nagata (#2)
Re: pgbench: extend variable usage in scripts

pá 29. 8. 2025 v 9:23 odesílatel Yugo Nagata <nagata@sraoss.co.jp> napsal:

Hi,

I would like to propose patches to extend variable usage in pgbench
scripts.

* 0001: Allow variables to be used as an SQL literal or identifier

Currently, variables in pgbench scripts are expanded always without
quotes, so they cannot be used as SQL literals or identifiers.
This patch allows the use of :'var' and :"var" in addition to :var

However, they can used only in SQL commands, not in the arguments of
meta-commands, since exprscan.l (the lexical scanner for pgbench backslash
commands) cannot currently handle quoted values.

Also, we have to use \aset, \gset or -D option to assign a string to a
varialbe,
since pgbench's \set command cannot recognize text values in its arguments.

* 0002: Add syntax for variable exisitence check

Currently, pgbench does not support :{?var} syntax to check whether the
variable
is defined or not. This patch adds support for this syntax in meta-command
arguments.
This is useful for checking if \aset set the result to the variable or not.

However, it cannot be used in SQL statements for now.

Some of the current limitations described above might be relaxed in the
future, but
for now, I would like to ask for initial feedback on this proposal.

+1

Regards

Pavel

Show quoted text

Regards,
Yugo Nagata

--
Yugo Nagata <nagata@sraoss.co.jp>

#4Yugo Nagata
nagata@sraoss.co.jp
In reply to: Pavel Stehule (#3)
Re: pgbench: extend variable usage in scripts

On Fri, 29 Aug 2025 09:27:36 +0200
Pavel Stehule <pavel.stehule@gmail.com> wrote:

pá 29. 8. 2025 v 9:23 odesílatel Yugo Nagata <nagata@sraoss.co.jp> napsal:

Hi,

I would like to propose patches to extend variable usage in pgbench
scripts.

* 0001: Allow variables to be used as an SQL literal or identifier

Currently, variables in pgbench scripts are expanded always without
quotes, so they cannot be used as SQL literals or identifiers.
This patch allows the use of :'var' and :"var" in addition to :var

While working on this patch, I was surprised to notice that variables are
expanded even inside quoted literals or identifiers in pgbench scripts,
which differs from the behavior in psql.

For example, a variable var is expanded in the following query if it is
defined. Otherwise, if the variable is not defined, the query result is
unchanged:

SELECT 'var is :var';

This means that we don’t actually need the new syntax proposed in patch 0001
to allow variables to be used as SQL literals or identifiers.

However, it also means that users need to be careful when a script contains
text or identifiers such as table names or function names that include a colon,
since such text could be unexpectedly rewritten.

Even worse, when "extended" or "prepared" mode is used, scripts containing a colon
will fail because any word starting with a colon is replaced with a placeholder like
"$1". For example, the query above fails with the following error, regardless of whether
the variable is defined or not:

ERROR: bind message supplies 1 parameters, but prepared statement "" requires 0
STATEMENT: SELECT 'var is $1';

In my opinion, it is not worth fixing the expansion of variables inside quoted text,
since I have never heard any complaints about it, and changing it would break backward
compatibility.

However, it would be better to update the documentation to note that variables are expanded
in all parts of an SQL command, and that this behavior differs from psql scripts.

Any thought?

However, they can used only in SQL commands, not in the arguments of
meta-commands, since exprscan.l (the lexical scanner for pgbench backslash
commands) cannot currently handle quoted values.

Also, we have to use \aset, \gset or -D option to assign a string to a
varialbe,
since pgbench's \set command cannot recognize text values in its arguments.

* 0002: Add syntax for variable exisitence check

Currently, pgbench does not support :{?var} syntax to check whether the
variable
is defined or not. This patch adds support for this syntax in meta-command
arguments.
This is useful for checking if \aset set the result to the variable or not.

However, it cannot be used in SQL statements for now.

Some of the current limitations described above might be relaxed in the
future, but
for now, I would like to ask for initial feedback on this proposal.

+1

Regards

Pavel

Regards,
Yugo Nagata

--
Yugo Nagata <nagata@sraoss.co.jp>

--
Yugo Nagata <nagata@sraoss.co.jp>

#5Michael Paquier
michael@paquier.xyz
In reply to: Yugo Nagata (#4)
Re: pgbench: extend variable usage in scripts

On Thu, Sep 18, 2025 at 03:33:30PM +0900, Yugo Nagata wrote:

Any thought?

Note that the CI is complaining a bit. Relevant extract:
[22:51:36.908] pgbench.c: In function ‘replaceVariable’:
[22:51:36.908] pgbench.c:1990:9: error: enumeration value
‘PQUOTE_SHELL_ARG’ not handled in switch [-Werror=switch]
[22:51:36.908] 1990 | switch (quote)
--
Michael