log bind parameter values on error
Hello,
I'd like to propose a patch to log bind parameter values not only when
logging duration,
but also on error (timeout in particular) or in whatever situation the
statement normally gets logged.
This mostly could be useful when the request originator doesn't log them
either, so it's hard
to reproduce the problem.
Unfortunately, when enabled, the feature comes with some memory and CPU
overhead,
as we cannot convert certain values to text when in aborted transaction.
We potentially could do the trick with built-in types, but it would need
cautious work with composite types,
and also require more computation on the logging stage, which is a risk
of cascading errors.
Custom types still wouldn't be loggable, even as passed by client, which
would be not great.
So I decided to cache textual representations on bind stage,
which is especially easy if the client uses text protocol.
Best,
Alexey
Attachments:
log_parameters_v001.patchtext/x-patch; name=log_parameters_v001.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4a7121a..7b38aed 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6265,6 +6265,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters" xreflabel="log_parameters">
+ <term><varname>log_parameters</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/portalcmds.c b/src/backend/commands/portalcmds.c
index 5684997..ba431ce 100644
--- a/src/backend/commands/portalcmds.c
+++ b/src/backend/commands/portalcmds.c
@@ -94,7 +94,7 @@ PerformCursorOpen(DeclareCursorStmt *cstmt, ParamListInfo params,
/*
* Create a portal and copy the plan and queryString into its memory.
*/
- portal = CreatePortal(cstmt->portalname, false, false);
+ portal = CreatePortal(cstmt->portalname, false, false, false);
oldContext = MemoryContextSwitchTo(portal->portalContext);
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 6036b73..363c096 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -404,6 +404,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = num_params;
+ paramLI->hasTextValues = false;
i = 0;
foreach(l, exprstates)
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index fc7c605..6d047cd 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -921,6 +921,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
fcache->paramLI = paramLI;
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index ad72667..532a365 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -1302,7 +1302,7 @@ SPI_cursor_open_internal(const char *name, SPIPlanPtr plan,
else
{
/* In this path, error if portal of same name already exists */
- portal = CreatePortal(name, false, false);
+ portal = CreatePortal(name, false, false, false);
}
/* Copy the plan's query string into the portal */
@@ -2399,6 +2399,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
for (i = 0; i < nargs; i++)
{
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index 79197b1..3c3b152 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -31,6 +31,8 @@
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We don't copy textual representations here.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -53,6 +55,7 @@ copyParamList(ParamListInfo from)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = from->numParams;
+ retval->hasTextValues = false;
for (i = 0; i < from->numParams; i++)
{
@@ -229,6 +232,7 @@ RestoreParamList(char **start_address)
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nparams;
+ paramLI->hasTextValues = false;
for (i = 0; i < nparams; i++)
{
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 5ab7d3c..2ad56a2 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -183,7 +183,7 @@ static void forbidden_in_wal_sender(char firstchar);
static List *pg_rewrite_query(Query *query);
static bool check_log_statement(List *stmt_list);
static int errdetail_execute(List *raw_parsetree_list);
-static int errdetail_params(ParamListInfo params);
+static int errdetail_params();
static int errdetail_abort(void);
static int errdetail_recovery_conflict(void);
static void start_xact_command(void);
@@ -1154,7 +1154,7 @@ exec_simple_query(const char *query_string)
* Create unnamed portal to run the query or queries in. If there
* already is one, silently drop it.
*/
- portal = CreatePortal("", true, true);
+ portal = CreatePortal("", true, true, true);
/* Don't display the portal in pg_cursors */
portal->visible = false;
@@ -1690,9 +1690,9 @@ exec_bind_message(StringInfo input_message)
* if the unnamed portal is specified.
*/
if (portal_name[0] == '\0')
- portal = CreatePortal(portal_name, true, true);
+ portal = CreatePortal(portal_name, true, true, true);
else
- portal = CreatePortal(portal_name, false, false);
+ portal = CreatePortal(portal_name, false, false, true);
/*
* Prepare to copy stuff into the portal's memory context. We do all this
@@ -1731,6 +1731,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters;
+
params = (ParamListInfo) palloc(offsetof(ParamListInfoData, params) +
numParams * sizeof(ParamExternData));
/* we have static list of params, so no hooks needed */
@@ -1741,6 +1744,8 @@ exec_bind_message(StringInfo input_message)
params->parserSetup = NULL;
params->parserSetupArg = NULL;
params->numParams = numParams;
+ /* mark as not having text values before we have populated them all */
+ params->hasTextValues = false;
for (int paramno = 0; paramno < numParams; paramno++)
{
@@ -1807,9 +1812,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1835,6 +1862,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1859,10 +1902,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1936,13 +1991,14 @@ exec_bind_message(StringInfo input_message)
*portal_name ? portal_name : "",
psrc->query_string),
errhidestmt(true),
- errdetail_params(params)));
+ errdetail_params()));
break;
}
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ PortalClearCurrentTop(portal);
debug_query_string = NULL;
}
@@ -1961,7 +2017,6 @@ exec_execute_message(const char *portal_name, long max_rows)
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
const char *prepStmtName;
- ParamListInfo portalParams;
bool save_log_statement_stats = log_statement_stats;
bool is_xact_command;
bool execute_is_fetch;
@@ -1978,6 +2033,7 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
+ PortalSetCurrentTop(portal);
/*
* If the original query was a null string, just return
@@ -2005,12 +2061,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
@@ -2019,7 +2069,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2071,7 +2120,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params()));
was_logged = true;
}
@@ -2160,7 +2209,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params()));
break;
}
@@ -2304,63 +2353,20 @@ errdetail_execute(List *raw_parsetree_list)
* Add an errdetail() line showing bind-parameter data, if available.
*/
static int
-errdetail_params(ParamListInfo params)
+errdetail_params()
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = GetCurrentPortalBindParameters();
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index ca02aac..f7984c9 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,16 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt)
+ {
+ char *param_values = GetCurrentPortalBindParameters();
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2956,17 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters)
+ {
+ char *param_values = GetCurrentPortalBindParameters();
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6fe1939..097a17b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -464,6 +464,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1235,6 +1236,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 1fa02d2..7ce88a2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -527,6 +527,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters = off # log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/backend/utils/mmgr/portalmem.c b/src/backend/utils/mmgr/portalmem.c
index 2b014c8..2f1bcb2 100644
--- a/src/backend/utils/mmgr/portalmem.c
+++ b/src/backend/utils/mmgr/portalmem.c
@@ -24,6 +24,7 @@
#include "miscadmin.h"
#include "storage/ipc.h"
#include "utils/builtins.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/timestamp.h"
@@ -53,6 +54,7 @@ typedef struct portalhashent
static HTAB *PortalHashTable = NULL;
+
#define PortalHashTableLookup(NAME, PORTAL) \
do { \
PortalHashEnt *hentry; \
@@ -88,6 +90,8 @@ do { \
elog(WARNING, "trying to delete portal name that does not exist"); \
} while(0)
+Portal current_top_portal = NULL;
+
static MemoryContext TopPortalContext = NULL;
@@ -163,6 +167,94 @@ PortalGetPrimaryStmt(Portal portal)
}
/*
+ * GetCurrentPortalBindParameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the current portal
+ * or no current portal, or the text representation of the parameters is not
+ * available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+GetCurrentPortalBindParameters()
+{
+ ParamListInfo params;
+ StringInfoData param_str;
+
+ /* Fail if no current portal */
+ if (!PortalIsValid(current_top_portal))
+ return NULL;
+
+ params = current_top_portal->portalParams;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
+/*
* CreatePortal
* Returns a new portal given a name.
*
@@ -170,9 +262,11 @@ PortalGetPrimaryStmt(Portal portal)
* same name (if false, an error is raised).
*
* dupSilent: if true, don't even emit a WARNING.
+ *
+ * markCurrent: mark as current top portal
*/
Portal
-CreatePortal(const char *name, bool allowDup, bool dupSilent)
+CreatePortal(const char *name, bool allowDup, bool dupSilent, bool markCurrent)
{
Portal portal;
@@ -223,6 +317,9 @@ CreatePortal(const char *name, bool allowDup, bool dupSilent)
/* reuse portal->name copy */
MemoryContextSetIdentifier(portal->portalContext, portal->name);
+ if (markCurrent)
+ PortalSetCurrentTop(portal);
+
return portal;
}
@@ -246,7 +343,7 @@ CreateNewPortal(void)
break;
}
- return CreatePortal(portalname, false, false);
+ return CreatePortal(portalname, false, false, false);
}
/*
@@ -458,6 +555,28 @@ MarkPortalFailed(Portal portal)
}
/*
+ * PortalSetCurrentTop
+ * mark a portal as the current one.
+ */
+void
+PortalSetCurrentTop(Portal portal)
+{
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+}
+
+/*
+ * PortalClearCurrentTop
+ * mark a portal as no longer the current one.
+ */
+void
+PortalClearCurrentTop(Portal portal)
+{
+ Assert(current_top_portal == portal);
+ current_top_portal = NULL;
+}
+
+/*
* PortalDrop
* Destroy the portal.
*/
@@ -508,6 +627,9 @@ PortalDrop(Portal portal, bool isTopCommit)
*/
PortalHashTableDelete(portal);
+ if (portal == current_top_portal)
+ current_top_portal = NULL;
+
/* drop cached plan reference, if any */
PortalReleaseCachedPlan(portal);
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index 04b03c7..c441240 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 64457c7..2151807 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/include/utils/portal.h b/src/include/utils/portal.h
index e4929b9..bcf74e7 100644
--- a/src/include/utils/portal.h
+++ b/src/include/utils/portal.h
@@ -200,6 +200,11 @@ typedef struct PortalData
*/
#define PortalIsValid(p) PointerIsValid(p)
+/*
+ * The top-level portal that the client is explicitly working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+extern PGDLLIMPORT Portal current_top_portal;
/* Prototypes for functions in utils/mmgr/portalmem.c */
extern void EnablePortalManager(void);
@@ -215,13 +220,18 @@ extern void AtSubAbort_Portals(SubTransactionId mySubid,
ResourceOwner myXactOwner,
ResourceOwner parentXactOwner);
extern void AtSubCleanup_Portals(SubTransactionId mySubid);
-extern Portal CreatePortal(const char *name, bool allowDup, bool dupSilent);
+extern Portal CreatePortal(const char *name,
+ bool allowDup,
+ bool dupSilent,
+ bool markCurrent);
extern Portal CreateNewPortal(void);
extern void PinPortal(Portal portal);
extern void UnpinPortal(Portal portal);
extern void MarkPortalActive(Portal portal);
extern void MarkPortalDone(Portal portal);
extern void MarkPortalFailed(Portal portal);
+extern void PortalSetCurrentTop(Portal portal);
+extern void PortalClearCurrentTop(Portal portal);
extern void PortalDrop(Portal portal, bool isTopCommit);
extern Portal GetPortalByName(const char *name);
extern void PortalDefineQuery(Portal portal,
@@ -231,6 +241,7 @@ extern void PortalDefineQuery(Portal portal,
List *stmts,
CachedPlan *cplan);
extern PlannedStmt *PortalGetPrimaryStmt(Portal portal);
+extern char *GetCurrentPortalBindParameters();
extern void PortalCreateHoldStore(Portal portal);
extern void PortalHashTableDeleteAll(void);
extern bool ThereAreNoReadyPortals(void);
On 15/12/2018 00:04, Alexey Bashtanov wrote:
I'd like to propose a patch to log bind parameter values not only when
logging duration,
but also on error (timeout in particular) or in whatever situation the
statement normally gets logged.
This mostly could be useful when the request originator doesn't log them
either, so it's hard
to reproduce the problem.
That's a reasonable problem to solve.
So I decided to cache textual representations on bind stage,
which is especially easy if the client uses text protocol.
That sounds like a plausible approach. Have you done any performance
measurements?
In your patch, I would organize the changes to the portal API a bit
differently. Don't change the signature of CreatePortal(). Get rid of
PortalSetCurrentTop() and PortalClearCurrentTop(). Just have a global
variable CurrentPortal and set it directly. And then change
GetCurrentPortalBindParameters() to take a Portal as argument. This can
all happen inside postgres.c without changing the Portal APIs.
In fact, maybe don't use the Portal structure at all and just store the
saved textualized values inside postgres.c in a static variable.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
That sounds like a plausible approach. Have you done any performance
measurements?
No I haven't yet
In your patch, I would organize the changes to the portal API a bit
differently. Don't change the signature of CreatePortal().
okay
Get rid of PortalSetCurrentTop() and PortalClearCurrentTop().
I'll remove them from Portal API, but possibly have them in postgres.c
if you don't mind, just to avoid code duplication.
Renamed and maybe even inlined.
Just have a global
variable CurrentPortal and set it directly. And then change
GetCurrentPortalBindParameters() to take a Portal as argument. This can
all happen inside postgres.c without changing the Portal APIs.
Okay, will do
In fact, maybe don't use the Portal structure at all and just store the
saved textualized values inside postgres.c in a static variable.
Unlike SQL, parameters may spend much more memory, so I'd have them
in portal memory context to make sure the memory is released earlier
rather than later.
Tracking individual variable lifetime like we do with debug_query_string
sounds doable but a bit non-straightforward to me,
see e.g. the tricks we do with transaction commands.
Also, I'd like to avoid early forming of the error string, as we may
need to combine
them differently in future, e.g. when logging in various logging formats
or languages.
One-by-one pfree-ing doesn't look tempting either.
Do you think it would be acceptable to leave them cached in parameters
structure?
Best,
Alex
On 02/01/2019 23:53, Alexey Bashtanov wrote:
In fact, maybe don't use the Portal structure at all and just store the
saved textualized values inside postgres.c in a static variable.Unlike SQL, parameters may spend much more memory, so I'd have them
in portal memory context to make sure the memory is released earlier
rather than later.
Having them in the portal structure is different from having it in the
portal memory context. Although there is perhaps value in keeping them
together.
Do you think it would be acceptable to leave them cached in parameters
structure?
Let's see how it looks.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hello Peter,
Unlike SQL, parameters may spend much more memory, so I'd have them
in portal memory context to make sure the memory is released earlier
rather than later.Having them in the portal structure is different from having it in the
portal memory context. Although there is perhaps value in keeping them
together.
yeah, to avoid pointers to deallocated areas
Let's see how it looks.
please see attached
Best,
Alex
Attachments:
log_parameters_v002.patchtext/x-patch; name=log_parameters_v002.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f64402a..924a76c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6270,6 +6270,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters" xreflabel="log_parameters">
+ <term><varname>log_parameters</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index a98c836..9ee3954 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -404,6 +404,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = num_params;
+ paramLI->hasTextValues = false;
i = 0;
foreach(l, exprstates)
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index de41588..195e3c5 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -921,6 +921,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
fcache->paramLI = paramLI;
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 94a53e0..398f0e3 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2399,6 +2399,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
for (i = 0; i < nargs; i++)
{
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index a89a25e..5aebd428 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -31,6 +31,8 @@
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We don't copy textual representations here.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -53,6 +55,7 @@ copyParamList(ParamListInfo from)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = from->numParams;
+ retval->hasTextValues = false;
for (i = 0; i < from->numParams; i++)
{
@@ -229,6 +232,7 @@ RestoreParamList(char **start_address)
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nparams;
+ paramLI->hasTextValues = false;
for (i = 0; i < nparams; i++)
{
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0c0891b..1f81fba 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -183,7 +189,7 @@ static void forbidden_in_wal_sender(char firstchar);
static List *pg_rewrite_query(Query *query);
static bool check_log_statement(List *stmt_list);
static int errdetail_execute(List *raw_parsetree_list);
-static int errdetail_params(ParamListInfo params);
+static int errdetail_params(Portal portal);
static int errdetail_abort(void);
static int errdetail_recovery_conflict(void);
static void start_xact_command(void);
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters;
+
params = (ParamListInfo) palloc(offsetof(ParamListInfoData, params) +
numParams * sizeof(ParamExternData));
/* we have static list of params, so no hooks needed */
@@ -1741,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
params->parserSetup = NULL;
params->parserSetupArg = NULL;
params->numParams = numParams;
+ /* mark as not having text values before we have populated them all */
+ params->hasTextValues = false;
for (int paramno = 0; paramno < numParams; paramno++)
{
@@ -1807,9 +1821,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1835,6 +1871,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1859,10 +1911,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1936,13 +2000,14 @@ exec_bind_message(StringInfo input_message)
*portal_name ? portal_name : "",
psrc->query_string),
errhidestmt(true),
- errdetail_params(params)));
+ errdetail_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1961,7 +2026,6 @@ exec_execute_message(const char *portal_name, long max_rows)
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
const char *prepStmtName;
- ParamListInfo portalParams;
bool save_log_statement_stats = log_statement_stats;
bool is_xact_command;
bool execute_is_fetch;
@@ -1978,7 +2042,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -2005,21 +2068,22 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
+ /*
+ * We do it for non-xact commands only, as an xact command
+ * 1) shouldn't have any parameters to log;
+ * 2) may have the portal dropped early.
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2071,7 +2135,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params(portal)));
was_logged = true;
}
@@ -2160,13 +2224,14 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params(current_top_portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2304,63 +2369,20 @@ errdetail_execute(List *raw_parsetree_list)
* Add an errdetail() line showing bind-parameter data, if available.
*/
static int
-errdetail_params(ParamListInfo params)
+errdetail_params(Portal portal)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(portal);
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2700,94 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(Portal portal)
+{
+ ParamListInfo params;
+ StringInfoData param_str;
+
+ /* Fail if no current portal */
+ if (!PortalIsValid(portal))
+ return NULL;
+
+ params = portal->portalParams;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4141,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..6f4ab5a 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,16 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt)
+ {
+ char *param_values = get_portal_bind_parameters(current_top_portal);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2956,17 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters)
+ {
+ char *param_values = get_portal_bind_parameters(current_top_portal);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 7eda7fd..f072741 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -464,6 +464,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1235,6 +1236,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f7c1dee..6d8bdc0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -526,6 +526,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters = off # log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/backend/utils/mmgr/portalmem.c b/src/backend/utils/mmgr/portalmem.c
index a92b454..62107d8 100644
--- a/src/backend/utils/mmgr/portalmem.c
+++ b/src/backend/utils/mmgr/portalmem.c
@@ -24,6 +24,7 @@
#include "miscadmin.h"
#include "storage/ipc.h"
#include "utils/builtins.h"
+#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/snapmgr.h"
#include "utils/timestamp.h"
@@ -53,6 +54,7 @@ typedef struct portalhashent
static HTAB *PortalHashTable = NULL;
+
#define PortalHashTableLookup(NAME, PORTAL) \
do { \
PortalHashEnt *hentry; \
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index ded3b3a..3b850a4 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..c96ea32 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(Portal portal);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c07e7b9..82e9a04 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/include/utils/portal.h b/src/include/utils/portal.h
index bbef89e..d3b2724 100644
--- a/src/include/utils/portal.h
+++ b/src/include/utils/portal.h
@@ -200,7 +200,6 @@ typedef struct PortalData
*/
#define PortalIsValid(p) PointerIsValid(p)
-
/* Prototypes for functions in utils/mmgr/portalmem.c */
extern void EnablePortalManager(void);
extern bool PreCommit_Portals(bool isPrepare);
@@ -215,7 +214,9 @@ extern void AtSubAbort_Portals(SubTransactionId mySubid,
ResourceOwner myXactOwner,
ResourceOwner parentXactOwner);
extern void AtSubCleanup_Portals(SubTransactionId mySubid);
-extern Portal CreatePortal(const char *name, bool allowDup, bool dupSilent);
+extern Portal CreatePortal(const char *name,
+ bool allowDup,
+ bool dupSilent);
extern Portal CreateNewPortal(void);
extern void PinPortal(Portal portal);
extern void UnpinPortal(Portal portal);
@@ -231,6 +232,7 @@ extern void PortalDefineQuery(Portal portal,
List *stmts,
CachedPlan *cplan);
extern PlannedStmt *PortalGetPrimaryStmt(Portal portal);
+extern char *GetCurrentPortalBindParameters(Portal portal);
extern void PortalCreateHoldStore(Portal portal);
extern void PortalHashTableDeleteAll(void);
extern bool ThereAreNoReadyPortals(void);
please see attached
sorry, some unintended changes sneaked in, please see the corrected patch
Attachments:
log_parameters_v003.patchtext/x-patch; name=log_parameters_v003.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index f64402a..924a76c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6270,6 +6270,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters" xreflabel="log_parameters">
+ <term><varname>log_parameters</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index a98c836..9ee3954 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -404,6 +404,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = num_params;
+ paramLI->hasTextValues = false;
i = 0;
foreach(l, exprstates)
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index de41588..195e3c5 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -921,6 +921,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
fcache->paramLI = paramLI;
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 94a53e0..398f0e3 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2399,6 +2399,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
for (i = 0; i < nargs; i++)
{
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index a89a25e..5aebd428 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -31,6 +31,8 @@
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We don't copy textual representations here.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -53,6 +55,7 @@ copyParamList(ParamListInfo from)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = from->numParams;
+ retval->hasTextValues = false;
for (i = 0; i < from->numParams; i++)
{
@@ -229,6 +232,7 @@ RestoreParamList(char **start_address)
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nparams;
+ paramLI->hasTextValues = false;
for (i = 0; i < nparams; i++)
{
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0c0891b..1f81fba 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -183,7 +189,7 @@ static void forbidden_in_wal_sender(char firstchar);
static List *pg_rewrite_query(Query *query);
static bool check_log_statement(List *stmt_list);
static int errdetail_execute(List *raw_parsetree_list);
-static int errdetail_params(ParamListInfo params);
+static int errdetail_params(Portal portal);
static int errdetail_abort(void);
static int errdetail_recovery_conflict(void);
static void start_xact_command(void);
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters;
+
params = (ParamListInfo) palloc(offsetof(ParamListInfoData, params) +
numParams * sizeof(ParamExternData));
/* we have static list of params, so no hooks needed */
@@ -1741,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
params->parserSetup = NULL;
params->parserSetupArg = NULL;
params->numParams = numParams;
+ /* mark as not having text values before we have populated them all */
+ params->hasTextValues = false;
for (int paramno = 0; paramno < numParams; paramno++)
{
@@ -1807,9 +1821,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1835,6 +1871,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1859,10 +1911,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1936,13 +2000,14 @@ exec_bind_message(StringInfo input_message)
*portal_name ? portal_name : "",
psrc->query_string),
errhidestmt(true),
- errdetail_params(params)));
+ errdetail_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1961,7 +2026,6 @@ exec_execute_message(const char *portal_name, long max_rows)
char completionTag[COMPLETION_TAG_BUFSIZE];
const char *sourceText;
const char *prepStmtName;
- ParamListInfo portalParams;
bool save_log_statement_stats = log_statement_stats;
bool is_xact_command;
bool execute_is_fetch;
@@ -1978,7 +2042,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -2005,21 +2068,22 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
+ /*
+ * We do it for non-xact commands only, as an xact command
+ * 1) shouldn't have any parameters to log;
+ * 2) may have the portal dropped early.
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2071,7 +2135,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params(portal)));
was_logged = true;
}
@@ -2160,13 +2224,14 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params(current_top_portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2304,63 +2369,20 @@ errdetail_execute(List *raw_parsetree_list)
* Add an errdetail() line showing bind-parameter data, if available.
*/
static int
-errdetail_params(ParamListInfo params)
+errdetail_params(Portal portal)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(portal);
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2700,94 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(Portal portal)
+{
+ ParamListInfo params;
+ StringInfoData param_str;
+
+ /* Fail if no current portal */
+ if (!PortalIsValid(portal))
+ return NULL;
+
+ params = portal->portalParams;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4141,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..6f4ab5a 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,16 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt)
+ {
+ char *param_values = get_portal_bind_parameters(current_top_portal);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2956,17 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters)
+ {
+ char *param_values = get_portal_bind_parameters(current_top_portal);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 7eda7fd..f072741 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -464,6 +464,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1235,6 +1236,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index f7c1dee..6d8bdc0 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -526,6 +526,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters = off # log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index ded3b3a..3b850a4 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..c96ea32 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(Portal portal);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c07e7b9..82e9a04 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
There appears to be a problem with how this affects current logging
behavior.
I'm using
pgbench -M extended -S -T 10 bench
to test the extended protocol.
Unpatched, with log_statement=all, you get something like
LOG: execute <unnamed>: SELECT abalance FROM pgbench_accounts WHERE aid
= $1;
DETAIL: parameters: $1 = '30223'
With your patch, with log_statement=all and log_parameters=on, you get
the same, but with log_statement=all and log_parameters=off you get
LOG: execute <unnamed>: SELECT abalance FROM pgbench_accounts WHERE aid
= $1;
DETAIL: parameters: $1 = UNKNOWN TYPE
We should probably keep the existing parameter logging working as before.
This also raises the question of the new parameter name. Parameters are
already logged. So the name should perhaps be more like
log_parameters_on_error.
Some API notes on your patch: I think you can change
get_portal_bind_parameters() to take a ParamListInfo, since you're not
doing anything with the Portal other than grab the parameters. And that
would allow you to keep the signature of errdetail_params() unchanged.
I did some performance tests using the commands shown above and didn't
find any problems. Obviously the default pgbench stuff isn't very
parameter-intensive. Do you have tests with more and larger parameter
values?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi Peter,
With your patch, with log_statement=all and log_parameters=on, you get
the same, but with log_statement=all and log_parameters=off you getLOG: execute <unnamed>: SELECT abalance FROM pgbench_accounts WHERE aid
= $1;
DETAIL: parameters: $1 = UNKNOWN TYPE
Thanks for spotting this, I've fixed it, see the new patch attached.
This also raises the question of the new parameter name. Parameters are
already logged. So the name should perhaps be more like
log_parameters_on_error.
Done
Some API notes on your patch: I think you can change
get_portal_bind_parameters() to take a ParamListInfo, since you're not
doing anything with the Portal other than grab the parameters. And that
would allow you to keep the signature of errdetail_params() unchanged.
Done
I did some performance tests using the commands shown above and didn't
find any problems. Obviously the default pgbench stuff isn't very
parameter-intensive. Do you have tests with more and larger parameter
values?
I've done some tests, but they are not very reproducible:
the difference between runs is more than the difference between master
vs feature branch
and log_parameters_on_error on vs off.
I was using a small java app, it tested the speed using only a single
connection.
See its code and the results attached.
I'm sorry for the delay, feel free to move it to next commitfest if needed.
Best,
Alex
Attachments:
log_parameters_v005.patchtext/x-patch; name=log_parameters_v005.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index b6f5822..997e6e8 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6274,6 +6274,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index a98c836..9ee3954 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -404,6 +404,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = num_params;
+ paramLI->hasTextValues = false;
i = 0;
foreach(l, exprstates)
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index c6b7203..e107fa3 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -921,6 +921,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
fcache->paramLI = paramLI;
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 94a53e0..398f0e3 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2399,6 +2399,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
for (i = 0; i < nargs; i++)
{
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index a89a25e..5aebd428 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -31,6 +31,8 @@
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We don't copy textual representations here.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -53,6 +55,7 @@ copyParamList(ParamListInfo from)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = from->numParams;
+ retval->hasTextValues = false;
for (i = 0; i < from->numParams; i++)
{
@@ -229,6 +232,7 @@ RestoreParamList(char **start_address)
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nparams;
+ paramLI->hasTextValues = false;
for (i = 0; i < nparams; i++)
{
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e773f20..69fa610 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = (ParamListInfo) palloc(offsetof(ParamListInfoData, params) +
numParams * sizeof(ParamExternData));
/* we have static list of params, so no hooks needed */
@@ -1741,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
params->parserSetup = NULL;
params->parserSetupArg = NULL;
params->numParams = numParams;
+ /* mark as not having text values before we have populated them all */
+ params->hasTextValues = false;
for (int paramno = 0; paramno < numParams; paramno++)
{
@@ -1807,9 +1821,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1835,6 +1871,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1859,10 +1911,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1943,6 +2007,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1978,7 +2043,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -2000,26 +2064,30 @@ exec_execute_message(const char *portal_name, long max_rows)
*/
if (is_xact_command)
{
+ portalParams = portal->portalParams;
+
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
+ /*
+ * We do it for non-xact commands only, as an xact command
+ * 1) shouldn't have any parameters to log;
+ * 2) may have the portal dropped early.
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+ portalParams = NULL;
+
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2160,13 +2228,14 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_params(current_top_portal->portalParams)));
break;
}
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2306,61 +2375,18 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(params);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2704,89 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ elog(WARNING, "params->hasTextValues=%d, IsAbortedTransactionBlockState()=%d",
+ params->hasTextValues && IsAbortedTransactionBlockState());
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4140,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..e18d9e5 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,18 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error
+ && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2958,18 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index c216ed0..892ca77 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -465,6 +465,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1242,6 +1243,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a21865a..b2865d8 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -526,6 +526,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index ded3b3a..3b850a4 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..2cfa2ac 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c07e7b9..1034fc2 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
On Mon, Jan 28, 2019 at 12:15:51AM +0000, Alexey Bashtanov wrote:
I'm sorry for the delay, feel free to move it to next commitfest if
needed.
Done.
--
Michael
This piece in your patch probably doesn't belong:
+ elog(WARNING, "params->hasTextValues=%d,
IsAbortedTransactionBlockState()=%d",
+ params->hasTextValues &&
IsAbortedTransactionBlockState());
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi,
On 2018-12-14 23:04:26 +0000, Alexey Bashtanov wrote:
Unfortunately, when enabled, the feature comes with some memory and CPU
overhead,
as we cannot convert certain values to text when in aborted transaction.
Have you analyzed how invasive it'd be to delay that till we actually
can safely start a [sub]transaction to do that logging? Probably too
expensive, but it seems like something that ought to be analyzed.
- Andres
Hi,
tiny scroll-through review.
On 2019-01-28 00:15:51 +0000, Alexey Bashtanov wrote:
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index b6f5822..997e6e8 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -6274,6 +6274,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a ' </listitem> </varlistentry>+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error"> + <term><varname>log_parameters_on_error</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>log_parameters_on_error</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Controls whether the statement is logged with bind parameter values. + It adds some overhead, as postgres will cache textual + representations of parameter values in memory for all statements, + even if they eventually do not get logged. The default is + <literal>off</literal>. Only superusers can change this setting. + </para> + </listitem> + </varlistentry>
This needs a bit of language polishing.
@@ -31,6 +31,8 @@ * set of parameter values. If dynamic parameter hooks are present, we * intentionally do not copy them into the result. Rather, we forcibly * instantiate all available parameter values and copy the datum values. + * + * We don't copy textual representations here. */
That probably needs to be expanded on. Including a comment about what
guarantees that there are no memory lifetime issues.
- /* Free result of encoding conversion, if any */ - if (pstring && pstring != pbuf.data) - pfree(pstring); + if (pstring) + { + if (need_text_values) + { + if (pstring == pbuf.data) + { + /* + * Copy textual representation to portal context. + */ + params->params[paramno].textValue = + pstrdup(pstring); + } + else + { + /* Reuse the result of encoding conversion for it */ + params->params[paramno].textValue = pstring; + } + } + else + { + /* Free result of encoding conversion */ + if (pstring != pbuf.data) + pfree(pstring); + } + }
So the parameters we log here haven't actually gone through the output
function? Isn't that an issue? I mean that'll cause the contents to
differ from the non-error case, no? And differs from the binary protocol
case?
else { + /* + * We do it for non-xact commands only, as an xact command + * 1) shouldn't have any parameters to log; + * 2) may have the portal dropped early. + */ + Assert(current_top_portal == NULL); + current_top_portal = portal; + portalParams = NULL; +
"it"? ISTM the comment doesn't really stand on its own?
+/* + * get_portal_bind_parameters + * Get the string containing parameters data, is used for logging. + * + * Can return NULL if there are no parameters in the portal + * or the portal is not valid, or the text representations of the parameters are + * not available. If returning a non-NULL value, it allocates memory + * for the returned string in the current context, and it's the caller's + * responsibility to pfree() it if needed. + */ +char * +get_portal_bind_parameters(ParamListInfo params) +{ + StringInfoData param_str; + + /* No parameters to format */ + if (!params || params->numParams == 0) + return NULL; + + elog(WARNING, "params->hasTextValues=%d, IsAbortedTransactionBlockState()=%d", + params->hasTextValues && IsAbortedTransactionBlockState());
Err, huh?
Greetings,
Andres Freund
Hello Anders and Peter,
Thanks for your messages.
Please see the new patch version attached.
Have you analyzed how invasive it'd be to delay that till we actually
can safely start a [sub]transaction to do that logging? Probably too
expensive, but it seems like something that ought to be analyzed.
The fundamental problem here is that the output functions for the types
of the values to be logged may be present only in the transaction
that has just been aborted.
Also I don't like the idea of doing complex operations in the error handler,
as it increases the chances of cascading errors.
I thought of pre-calculating types' output functions instead of their
results,
but that would work for certain built-in types only.
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error"> + <term><varname>log_parameters_on_error</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>log_parameters_on_error</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Controls whether the statement is logged with bind parameter values. + It adds some overhead, as postgres will cache textual + representations of parameter values in memory for all statements, + even if they eventually do not get logged. The default is + <literal>off</literal>. Only superusers can change this setting. + </para> + </listitem> + </varlistentry>This needs a bit of language polishing.
I would appreciate if you had any suggestions, as my English isn't great.
@@ -31,6 +31,8 @@ * set of parameter values. If dynamic parameter hooks are present, we * intentionally do not copy them into the result. Rather, we forcibly * instantiate all available parameter values and copy the datum values. + * + * We don't copy textual representations here. */That probably needs to be expanded on. Including a comment about what
guarantees that there are no memory lifetime issues.
What kind of memory lifetime issues do you mean?
We're not copying textual representations, so the worst can happen
is they don't get logged when appropriate. Luckily, we never use this
function when copying to a portal we use for logging, I added this to
the comment. Do you think it's any better?
So the parameters we log here haven't actually gone through the output
function? Isn't that an issue? I mean that'll cause the contents to
differ from the non-error case, no? And differs from the binary protocol
case?
I don't think it's much of a problem. Text input and output functions are
meant to match, but the CREATE TYPE spec isn't too specific about what
it means.
Of course it does not mean that typoutput(typinput(foo)) is always
exactly foo.
However, I really hope that at least typinput(typoutput(foo)) = foo,
where "=" is the correspondent operator registered in postgres.
If a cheeky client passes '007' as a numeric value I don't mind it being
sometimes logged as '007' and sometimes as '7', depending on the settings.
It anyway denotes the same number, and we'll know what to pass to
reproduce the problem.
For binary protocol it'll be '7' as well, as it'll undergo the typrecv
and then typoutput procedures.
else { + /* + * We do it for non-xact commands only, as an xact command + * 1) shouldn't have any parameters to log; + * 2) may have the portal dropped early. + */ + Assert(current_top_portal == NULL); + current_top_portal = portal; + portalParams = NULL; +"it"? ISTM the comment doesn't really stand on its own?
Thanks, I fixed the comment and some code around it.
+/* + * get_portal_bind_parameters + * Get the string containing parameters data, is used for logging. + * + * Can return NULL if there are no parameters in the portal + * or the portal is not valid, or the text representations of the parameters are + * not available. If returning a non-NULL value, it allocates memory + * for the returned string in the current context, and it's the caller's + * responsibility to pfree() it if needed. + */ +char * +get_portal_bind_parameters(ParamListInfo params) +{ + StringInfoData param_str; + + /* No parameters to format */ + if (!params || params->numParams == 0) + return NULL; + + elog(WARNING, "params->hasTextValues=%d, IsAbortedTransactionBlockState()=%d", + params->hasTextValues && IsAbortedTransactionBlockState());Err, huh?
This was some debugging, I threw it away now.
Best,
Alex
Attachments:
log_parameters_v006.patchtext/x-patch; name=log_parameters_v006.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8bd57f3..59638f8 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6312,6 +6312,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index a98c836..9ee3954 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -404,6 +404,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = num_params;
+ paramLI->hasTextValues = false;
i = 0;
foreach(l, exprstates)
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index c6b7203..e107fa3 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -921,6 +921,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
fcache->paramLI = paramLI;
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 70c03e0..019cef7 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2399,6 +2399,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nargs;
+ paramLI->hasTextValues = false;
for (i = 0; i < nargs; i++)
{
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index a89a25e..72d7dd1 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -31,6 +31,10 @@
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * Since this function is never used for copying parameters into a top portal,
+ * we don't copy textual representations. Neither we set the hasTextValues
+ * flag, so noone will access them.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -53,6 +57,7 @@ copyParamList(ParamListInfo from)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = from->numParams;
+ retval->hasTextValues = false;
for (i = 0; i < from->numParams; i++)
{
@@ -229,6 +234,7 @@ RestoreParamList(char **start_address)
paramLI->parserSetup = NULL;
paramLI->parserSetupArg = NULL;
paramLI->numParams = nparams;
+ paramLI->hasTextValues = false;
for (i = 0; i < nparams; i++)
{
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8b4d94c..38df7ca 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = (ParamListInfo) palloc(offsetof(ParamListInfoData, params) +
numParams * sizeof(ParamExternData));
/* we have static list of params, so no hooks needed */
@@ -1741,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
params->parserSetup = NULL;
params->parserSetupArg = NULL;
params->numParams = numParams;
+ /* mark as not having text values before we have populated them all */
+ params->hasTextValues = false;
for (int paramno = 0; paramno < numParams; paramno++)
{
@@ -1807,9 +1821,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1835,6 +1871,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1859,10 +1911,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1943,6 +2007,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1978,7 +2043,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1993,32 +2057,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2167,6 +2248,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2306,61 +2388,18 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(params);
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2717,87 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4151,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..e18d9e5 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,18 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error
+ && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2958,18 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 156d147..a83fe3c 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -480,6 +480,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1257,6 +1258,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 194f312..e8f74a2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -528,6 +528,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index ded3b3a..3b850a4 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..2cfa2ac 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index c07e7b9..1034fc2 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
On 2019-02-15 15:02, Alexey Bashtanov wrote:
Hello Anders and Peter,
Thanks for your messages.
Please see the new patch version attached.
In my testing, I couldn't get this patch to do anything. Could you
please share your testing steps?
I did
postgres -D data --log-parameters-on-error=on
pgbench -i bench
alter table pgbench_accounts alter column aid type smallint;
pgbench -M extended -S -T 10 bench
This will then error out on type overflows, but I don't see any
parameters being logged:
ERROR: value "62812" is out of range for type smallint
STATEMENT: SELECT abalance FROM pgbench_accounts WHERE aid = $1;
(In this case the error message contains the parameter value, so it's
not a very practical case, but it should work, it seems.)
Meanwhile, I have committed a patch that refactors the ParamListInfo
initialization a bit, so you don't have to initialize hasTextValues in a
bunch of places unrelated to your core code. So please rebase your
patch on that.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi Alexey,
On 3/14/19 12:38 PM, Peter Eisentraut wrote:
Meanwhile, I have committed a patch that refactors the ParamListInfo
initialization a bit, so you don't have to initialize hasTextValues in a
bunch of places unrelated to your core code. So please rebase your
patch on that.
It's been two weeks with no new patch or answers to Peter's questions.
Since we are nearly at the end of the CF I'll target this patch for PG13
and mark it Returned with Feedback at the end of the CF if there's no
new patch by then.
Regards,
--
-David
david@pgmasters.net
Hello and sorry for weeks of silence.
Hello Anders and Peter,
Thanks for your messages.
Please see the new patch version attached.In my testing, I couldn't get this patch to do anything. Could you
please share your testing steps?
Sure. Provided you're in the postgres checkout and you've run make in
src/test/examples/ this should work
CREATE SCHEMA testlibpq3;
SET search_path = testlibpq3;
CREATE TABLE test1_(i int4, t text, b bytea);
INSERT INTO test1_ VALUES(0, '', '');
CREATE VIEW test1 AS SELECT 1/i i, t, b FROM test1_;
-- will log only statement
\! ./src/test/examples/testlibpq3
ALTER SYSTEM SET log_parameters_on_error TO on;
SELECT pg_reload_conf();
-- will log statement with parameters
\! ./src/test/examples/testlibpq3
I did
postgres -D data --log-parameters-on-error=on
pgbench -i bench
alter table pgbench_accounts alter column aid type smallint;
pgbench -M extended -S -T 10 bench
This will then error out on type overflows, but I don't see any
parameters being logged:ERROR: value "62812" is out of range for type smallint
STATEMENT: SELECT abalance FROM pgbench_accounts WHERE aid = $1;(In this case the error message contains the parameter value, so it's
not a very practical case, but it should work, it seems.)
I guess this error occurred /while/ binding, so the parameters probably
weren't yet all bound by the time of error reporting.
That's why the error message came without parameters.
Meanwhile, I have committed a patch that refactors the ParamListInfo
initialization a bit, so you don't have to initialize hasTextValues in a
bunch of places unrelated to your core code. So please rebase your
patch on that.
Please find rebased patch attached.
I apologize for no reply before: I first thought my patch was really
faulty and knew I wouldn't have time to fix it these days, but it seems
to me it actually works.
Anyway, I don't suppose you manage to review it within the remaining few
days, so I'll rebase it again in the beginning of the next CF.
Best regards,
Alexey
Attachments:
log_parameters_v007.patchtext/x-patch; name=log_parameters_v007.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d383de2..1442f30 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6330,6 +6330,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index f5d5613..5e5ccf1 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -58,6 +59,10 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * Since this function is never used for copying parameters into a top portal,
+ * we don't copy textual representations. Neither we set the hasTextValues
+ * flag, so noone will access them.
*/
ParamListInfo
copyParamList(ParamListInfo from)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f9ce3d8..e097ce8 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1798,9 +1810,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1826,6 +1860,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1850,10 +1900,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * only now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1934,6 +1996,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1969,7 +2032,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1984,32 +2046,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2158,6 +2237,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2297,61 +2377,18 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(params);
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2669,6 +2706,87 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4022,10 +4140,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..e18d9e5 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,18 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error
+ && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2958,18 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index aa564d1..acd4f1e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -481,6 +481,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1254,6 +1255,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cccb5f1..61ec41a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -528,6 +528,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd90466..316d3a2 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..2cfa2ac 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 2712a77..563dcea 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -232,6 +232,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
On 2019-03-29 15:55, Alexey Bashtanov wrote:
ERROR: value "62812" is out of range for type smallint
STATEMENT: SELECT abalance FROM pgbench_accounts WHERE aid = $1;(In this case the error message contains the parameter value, so it's
not a very practical case, but it should work, it seems.)I guess this error occurred /while/ binding, so the parameters probably
weren't yet all bound by the time of error reporting.
That's why the error message came without parameters.
I see. But I think that could be fixed. Change exec_bind_message() to
loop over the parameters twice: once to save them away, once to actually
process them. I think the case of a faulty input value is probably very
common, so it would be confusing if that didn't work.
I think this patch needs some tests. Manually testing it is cumbersome,
and as we are seeing now, it is not quite clear which cases it is
supposed to cover. There are also additional cases for binary
parameters, and there are additions to the CSV output format. We need
tests for all that so the behavior explains itself and doesn't have to
be rediscovered every time someone wants to look at this again.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 05/04/2019 12:23, Peter Eisentraut wrote:
On 2019-03-29 15:55, Alexey Bashtanov wrote:
ERROR: value "62812" is out of range for type smallint
STATEMENT: SELECT abalance FROM pgbench_accounts WHERE aid = $1;(In this case the error message contains the parameter value, so it's
not a very practical case, but it should work, it seems.)I guess this error occurred /while/ binding, so the parameters probably
weren't yet all bound by the time of error reporting.
That's why the error message came without parameters.I see. But I think that could be fixed. Change exec_bind_message() to
loop over the parameters twice: once to save them away, once to actually
process them. I think the case of a faulty input value is probably very
common, so it would be confusing if that didn't work.
I understand this may sound lazy of me, however let me take this risk
and try to explain why I think logging them here in the same fashion
would be inconsistent.
The original feature is intended to log textual representations of
the bind parameter values, whereas the problem suggested to be solved
together with it is verbose logging of the part of the bind message that
failed to get converted to a Datum.
These are equivalent only under the following conditions:
1) the rest of the message was correctly formatted, so we can extract
something for each of the parameter values
2) the values have been passed in text mode
(all of them, not only the one failed to process)
3) the values have been passed in the server encoding
I think it would be a bit of inconsistent to log the parts of the message
only when we are lucky to satisfy the 3 conditions above.
If we are to log bind parameters message parsing and processing problems
more verbosely, what do you think of rather wrapping calls to
OidInputFunctionCall, OidReceiveFunctionCall and pg_client_to_server
into PG_TRY blocks and logging their arguments individually?
In case it doesn't work for any reason, I have an alternative idea on
how to log half-processed parameters.
Instead of hasTextValues store the number of parameters having
their textual representations successfully saved.
This way we will be able, in case of text mode,
save the original value and increment the counter
before the call to OidInputFunctionCall. When logging and not having values
for some parameters, we can print an ellipsis in the log to indicate there's
some more of them missing.
I think this patch needs some tests. Manually testing it is cumbersome,
and as we are seeing now, it is not quite clear which cases it is
supposed to cover. There are also additional cases for binary
parameters, and there are additions to the CSV output format. We need
tests for all that so the behavior explains itself and doesn't have to
be rediscovered every time someone wants to look at this again.
I have added a section into src/test/examples/testlibpq3.c,
please see attached.
As far as I could see these tests never run on "make check" or "make
installcheck",
hence the README change. Please correct me if I'm wrong.
I also failed to find an automatic way to test what actually gets logged
to the server log, at least not in the installcheck case.
I would appreciate if you had any suggestions about it.
Best regards,
Alex
Attachments:
log_parameters_v008.patchtext/x-patch; name=log_parameters_v008.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 75f9471..437dc44 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6418,6 +6418,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index f5d5613..5e5ccf1 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -58,6 +59,10 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * Since this function is never used for copying parameters into a top portal,
+ * we don't copy textual representations. Neither we set the hasTextValues
+ * flag, so noone will access them.
*/
ParamListInfo
copyParamList(ParamListInfo from)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 44a59e1..3559163 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1798,9 +1810,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1826,6 +1860,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1850,10 +1900,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * only now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1934,6 +1996,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1969,7 +2032,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1984,32 +2046,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2158,6 +2237,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2299,61 +2379,18 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(params);
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2671,6 +2708,87 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4024,10 +4142,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720e..e18d9e5 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,18 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error
+ && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2958,18 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index f7f726b..9034303 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -483,6 +483,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1296,6 +1297,15 @@ static struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
+ {
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 77bb7c2..b21ca36 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -536,6 +536,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd90466..316d3a2 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index b367838..2cfa2ac 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -24,6 +24,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -32,6 +33,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -83,6 +86,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 6c41eda..e5f8d39 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0..22e624d 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c
index c3b524c..3b76d66 100644
--- a/src/test/examples/testlibpq3.c
+++ b/src/test/examples/testlibpq3.c
@@ -42,7 +42,6 @@
#include <netinet/in.h>
#include <arpa/inet.h>
-
static void
exit_nicely(PGconn *conn)
{
@@ -116,9 +115,9 @@ main(int argc, char **argv)
const char *conninfo;
PGconn *conn;
PGresult *res;
- const char *paramValues[1];
- int paramLengths[1];
- int paramFormats[1];
+ const char *paramValues[2];
+ int paramLengths[2];
+ int paramFormats[2];
uint32_t binaryIntVal;
/*
@@ -224,6 +223,52 @@ main(int argc, char **argv)
PQclear(res);
+ /*
+ * In the third example we transmit parameters in different forms, make
+ * a statement fail and check how logging parameters on error works.
+ */
+ /* Set always-secure search path, so malicious users can't take control. */
+
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ res = PQexecParams(conn,
+ /*
+ * always divide by zero,
+ * but deduce it only when executing
+ */
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int",
+ 2, /* two params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero expected, got an error message from server: %s"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
/* close the connection to the database and cleanup */
PQfinish(conn);
Please find the rebased patch attached.
Tested like the following.
Provided you're in the postgres checkout and you've run make in
src/test/examples/ and connected to db=postgres:
CREATE SCHEMA testlibpq3;
SET search_path = testlibpq3;
CREATE TABLE test1_(i int4, t text, b bytea);
INSERT INTO test1_ VALUES(0, '', '');
CREATE VIEW test1 AS SELECT 1/i i, t, b FROM test1_;
-- will log only statement
\! ./src/test/examples/testlibpq3
ALTER SYSTEM SET log_parameters_on_error TO on;
SELECT pg_reload_conf();
-- will log statement with parameters
\! ./src/test/examples/testlibpq3
Best regards,
Alexey
Attachments:
log_parameters_v009.patchtext/x-patch; name=log_parameters_v009.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c91e3e1550..9f2e9bae33 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6423,6 +6423,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index f5d56138ee..5e5ccf1a4f 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -58,6 +59,10 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * Since this function is never used for copying parameters into a top portal,
+ * we don't copy textual representations. Neither we set the hasTextValues
+ * flag, so noone will access them.
*/
ParamListInfo
copyParamList(ParamListInfo from)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 44a59e1d4f..3559163ac3 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1694,6 +1700,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1731,6 +1740,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1798,9 +1810,31 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ if (pstring)
+ {
+ if (need_text_values)
+ {
+ if (pstring == pbuf.data)
+ {
+ /*
+ * Copy textual representation to portal context.
+ */
+ params->params[paramno].textValue =
+ pstrdup(pstring);
+ }
+ else
+ {
+ /* Reuse the result of encoding conversion for it */
+ params->params[paramno].textValue = pstring;
+ }
+ }
+ else
+ {
+ /* Free result of encoding conversion */
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1826,6 +1860,22 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging. We waste
+ * some time and memory here, maybe one day we could skip
+ * certain types like built-in primitives, which are safe to get
+ * it calculated later in an aborted xact.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1850,10 +1900,22 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * only now we can safely set it, as we have textValue populated
+ * for all non-null parameters
+ */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1934,6 +1996,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -1969,7 +2032,6 @@ exec_execute_message(const char *portal_name, long max_rows)
ereport(ERROR,
(errcode(ERRCODE_UNDEFINED_CURSOR),
errmsg("portal \"%s\" does not exist", portal_name)));
-
/*
* If the original query was a null string, just return
* EmptyQueryResponse.
@@ -1984,32 +2046,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2158,6 +2237,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2299,61 +2379,18 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ MemoryContext oldcontext = MemoryContextSwitchTo(MessageContext);
+ char *params_message = get_portal_bind_parameters(params);
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2671,6 +2708,87 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_parameters
+ * Get the string containing parameters data, is used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal
+ * or the portal is not valid, or the text representations of the parameters are
+ * not available. If returning a non-NULL value, it allocates memory
+ * for the returned string in the current context, and it's the caller's
+ * responsibility to pfree() it if needed.
+ */
+char *
+get_portal_bind_parameters(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calcualted,
+ * or call potentially user-defined I/O functions to convert internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4024,10 +4142,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720ef3a..e18d9e59c8 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,18 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error
+ && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ appendCSVLiteral(&buf, param_values);
+ pfree(param_values);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2958,18 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *param_values =
+ get_portal_bind_parameters(current_top_portal->portalParams);
+ if (param_values != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), param_values);
+ pfree(param_values);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 631f16f5fe..2e0b265f4b 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -483,6 +483,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1294,6 +1295,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cfad86c02a..02e60b3bd2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -536,6 +536,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..316d3a2888 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index ec21f7e45c..ecd6a93e2c 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -19,6 +19,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -27,6 +28,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -78,6 +81,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_parameters(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index e709177c37..125dc13a31 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/testlibpq3.c b/src/test/examples/testlibpq3.c
index c3b524cfdf..3b76d66e7d 100644
--- a/src/test/examples/testlibpq3.c
+++ b/src/test/examples/testlibpq3.c
@@ -42,7 +42,6 @@
#include <netinet/in.h>
#include <arpa/inet.h>
-
static void
exit_nicely(PGconn *conn)
{
@@ -116,9 +115,9 @@ main(int argc, char **argv)
const char *conninfo;
PGconn *conn;
PGresult *res;
- const char *paramValues[1];
- int paramLengths[1];
- int paramFormats[1];
+ const char *paramValues[2];
+ int paramLengths[2];
+ int paramFormats[2];
uint32_t binaryIntVal;
/*
@@ -224,6 +223,52 @@ main(int argc, char **argv)
PQclear(res);
+ /*
+ * In the third example we transmit parameters in different forms, make
+ * a statement fail and check how logging parameters on error works.
+ */
+ /* Set always-secure search path, so malicious users can't take control. */
+
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ res = PQexecParams(conn,
+ /*
+ * always divide by zero,
+ * but deduce it only when executing
+ */
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int",
+ 2, /* two params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero expected, got an error message from server: %s"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
/* close the connection to the database and cleanup */
PQfinish(conn);
Nice patch, thanks.
I didn't like abusing testlibpq3.c for your new stuff, so I moved it off
to a new file testlibpq5.c. I cleaned up a few other cosmetics things
about this -- v10 attached. I eventually noticed that this patch fails
to initialize each param's textValue to NULL, which probably explains
why you have to be so careful about only setting hasTextValues after the
whole loop. That seems a bit too trusting; I think it would be better
to set all these to NULL in makeParamList instead of leaving the memory
undefined. One way would be to have a for() look in makeParamList that
nullifies the member; another would be to use palloc0().
A third possibility is to inspect each caller of makeParamList and have
them all set textValue to NULL to each parameter.
I'm marking this waiting on author.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v10-0001-Allow-logging-of-portal-parameters-on-error.patchtext/x-diff; charset=us-asciiDownload
From 19c80d631694366e8098a564ba347fb1e937056b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 18 Sep 2019 16:56:42 -0300
Subject: [PATCH v10] Allow logging of portal parameters on error
---
doc/src/sgml/config.sgml | 17 ++
src/backend/nodes/params.c | 3 +
src/backend/tcop/postgres.c | 233 +++++++++++++-----
src/backend/utils/error/elog.c | 28 +++
src/backend/utils/misc/guc.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/params.h | 3 +
src/include/tcop/tcopprot.h | 4 +
src/include/utils/guc.h | 1 +
src/test/README | 3 +-
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpq5.c | 116 +++++++++
12 files changed, 355 insertions(+), 66 deletions(-)
create mode 100644 src/test/examples/testlibpq5.c
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6612f95f9f..81dfee5fe7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6414,6 +6414,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..00ad6347c7 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -58,6 +59,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We don't bother copying text values, since no caller needs that at present.
*/
ParamListInfo
copyParamList(ParamListInfo from)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e8d8e6f828..ec1e4092f5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1714,6 +1720,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1751,6 +1760,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1818,9 +1830,18 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * If we need the parameter string, keep it -- possibly making
+ * a copy first, if necessary. Otherwise get rid of it.
+ */
+ if (pstring)
+ {
+ if (need_text_values)
+ params->params[paramno].textValue =
+ pstring == pbuf.data ? pstrdup(pstring) : pstring;
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1846,6 +1867,23 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging.
+ *
+ * This could be optimized for certain built-in types, for
+ * which the output function is safe to call even in an
+ * aborted transaction.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1870,10 +1908,19 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /* Only now can we safely set this. */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1954,6 +2001,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2004,32 +2052,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2178,6 +2243,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2306,61 +2372,21 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ MemoryContext oldcontext;
+ char *params_message;
+
+ /* Make sure any trash is generated in MessageContext */
+ oldcontext = MemoryContextSwitchTo(MessageContext);
+ params_message = get_portal_bind_params(params);
+
+ if (params_message)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2704,86 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_params
+ * Get a string containing parameters data -- used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal or the portal is
+ * not valid, or the text representations of the parameters are not available.
+ * If returning a non-NULL value, it allocates memory for the returned string
+ * in the current context, and it's the caller's responsibility to pfree() it.
+ */
+char *
+get_portal_bind_params(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calculated, or
+ * call potentially user-defined I/O functions to convert the internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4137,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720ef3a..40896a2d09 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,19 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error &&
+ PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ appendCSVLiteral(&buf, params);
+ pfree(params);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2959,19 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), params);
+ pfree(params);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 90ffd89339..d426272be3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -483,6 +483,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1293,6 +1294,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3a61..fccfff3a03 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -532,6 +532,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..316d3a2888 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index ec21f7e45c..ad6517414b 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -19,6 +19,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -27,6 +28,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -78,6 +81,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_params(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 6791e0cbc2..57acc28ef9 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..69b65c512b
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,116 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ /* divide by zero -- but server won't realize until execution */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero expected, got an error message from server: %s\n"
+ "Please make sure it has been logged with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.17.1
Hi,
On 2019-09-18 17:58:53 -0300, Alvaro Herrera wrote:
--- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -6414,6 +6414,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a ' </listitem> </varlistentry>+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error"> + <term><varname>log_parameters_on_error</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>log_parameters_on_error</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Controls whether the statement is logged with bind parameter values.
Trailing whitespace.
I find the "the statement" formulation a bit odd, but can't quite put my
finger on why.
I think it might be worthwhile to cross-reference
log_min_error_statement, as log_parameters_on_error will only take effect when the
statement is logged due to log_min_error_statement.
+ It adds some overhead, as postgres will cache textual + representations of parameter values in memory for all statements, + even if they eventually do not get logged. The default is + <literal>off</literal>. Only superusers can change this setting.
I don't think "cache" is the right descriptor. Usually caching implies
trying to make repeated tasks faster, which isn't the case here.
+/* + * The top-level portal that the client is immediately working with: + * creating, binding, executing, or all at one using simple protocol + */ +Portal current_top_portal = NULL; +
This strikes me as decidedly not nice. For one this means that we'll not
be able to use this infrastructure for binds that are done serverside,
e.g. as part of plpgsql. I'm basically inclined to think that
integrating this at the postges.c level is the wrong thing.
It also adds new error handling complexity, which is already quite
substantial. And spreads knowledge of portals to elog.c, which imo
shouldn't have to know about them.
It seems to me that this would really need to be tracked inside the
portal infrastructure. To avoid unnecessary overhead, we could continue
to set the text values in exec_bin_message() in the pformat == 0 case,
using hasTextValues somewhere in the portal code to determine whether
the text representation has to be computed (for other input format, and
internal queries as e.g. generated by plpgsql).
And then the PG_CATCHes in pquery.c can add the errdetail() in the error
cases using an econtext callback.
+/* + * get_portal_bind_params + * Get a string containing parameters data -- used for logging. + * + * Can return NULL if there are no parameters in the portal or the portal is + * not valid, or the text representations of the parameters are not available. + * If returning a non-NULL value, it allocates memory for the returned string + * in the current context, and it's the caller's responsibility to pfree() it. + */ +char * +get_portal_bind_params(ParamListInfo params) +{ + StringInfoData param_str; + + /* No parameters to format */ + if (!params || params->numParams == 0) + return NULL; + + /* + * We either need textual representation of parameters pre-calculated, or + * call potentially user-defined I/O functions to convert the internal + * representation into text, which cannot be done in an aborted xact + */ + if (!params->hasTextValues && IsAbortedTransactionBlockState()) + return NULL;
Maybe I'm naive, but what is the point of keeping the separate
parameters, allocated separately, when all we're doing is building a
string containing them all at error time? Seems better just directly
form the full string when decideding to keep the text parameters - for
one it'll often end up being more efficient. But more importantly it
also makes it a lot less likely to run out of memory while handling the
error. The individual text parameters can be large, and this will always
additionally need at least the combined size of all parameters from
within the error context. That's not great.
+ appendStringInfoCharMacro(¶m_str, '\''); + for (p = pstring; *p; p++) + { + if (*p == '\'') /* double single quotes */ + appendStringInfoCharMacro(¶m_str, *p); + appendStringInfoCharMacro(¶m_str, *p); + } + appendStringInfoCharMacro(¶m_str, '\'');
I know this is old code, but: This is really inefficient. Will cause a
lot of unnecessary memory-reallocations for large text outputs, because
we don't immediately grow to at least close to the required size.
Greetings,
Andres Freund
Hi, thanks for looking.
On 2019-Sep-20, Andres Freund wrote:
On 2019-09-18 17:58:53 -0300, Alvaro Herrera wrote:
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error"> + <term><varname>log_parameters_on_error</varname> (<type>boolean</type>) + <indexterm> + <primary><varname>log_parameters_on_error</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Controls whether the statement is logged with bind parameter values.Trailing whitespace.
I find the "the statement" formulation a bit odd, but can't quite put my
finger on why.
Yeah, I think that wording is pretty confusing. I would use "Controls
whether bind parameters are logged when a statement is logged."
+/* + * The top-level portal that the client is immediately working with: + * creating, binding, executing, or all at one using simple protocol + */ +Portal current_top_portal = NULL; +This strikes me as decidedly not nice. For one this means that we'll not
be able to use this infrastructure for binds that are done serverside,
e.g. as part of plpgsql. I'm basically inclined to think that
integrating this at the postges.c level is the wrong thing.
[...]
It seems to me that this would really need to be tracked inside the
portal infrastructure.
I think that's how this was done at first, then Peter E drove him away
from that into the current design.
It also adds new error handling complexity, which is already quite
substantial. And spreads knowledge of portals to elog.c, which imo
shouldn't have to know about them.
Makes sense.
+ appendStringInfoCharMacro(¶m_str, '\''); + for (p = pstring; *p; p++) + { + if (*p == '\'') /* double single quotes */ + appendStringInfoCharMacro(¶m_str, *p); + appendStringInfoCharMacro(¶m_str, *p); + } + appendStringInfoCharMacro(¶m_str, '\'');I know this is old code, but: This is really inefficient. Will cause a
lot of unnecessary memory-reallocations for large text outputs, because
we don't immediately grow to at least close to the required size.
Agreed, but we can't blame a patch because it moves around some old
crufty code. If Alexey wants to include another patch to change this to
a better formulation, I'm happy to push that before or after his main
patch. And if he doesn't want to, that's fine with me too.
(Is doing a strlen first to enlarge the stringinfo an overall better
approach?) (I wonder if it would make sense to strchr each ' and memcpy
the intervening bytes ... if only that didn't break the SI abstraction
completely ...)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi,
On 2019-09-20 16:56:57 -0300, Alvaro Herrera wrote:
+/* + * The top-level portal that the client is immediately working with: + * creating, binding, executing, or all at one using simple protocol + */ +Portal current_top_portal = NULL; +This strikes me as decidedly not nice. For one this means that we'll not
be able to use this infrastructure for binds that are done serverside,
e.g. as part of plpgsql. I'm basically inclined to think that
integrating this at the postges.c level is the wrong thing.[...]
It seems to me that this would really need to be tracked inside the
portal infrastructure.I think that's how this was done at first, then Peter E drove him away
from that into the current design.
I don't think it really was the way I am suggesting. There were a bunch
of helper functions managing current_top_portal, but it otherwise is
(and afaict was in all versions) still all postgres.c
controlled. Whereas I think it should be nearly exclusively be handled
by pquery.c, with the possible exception of an efficiency hack to reuse
client input string when they're in text format.
What I'm suggesting is that PortalStart() would build a string
representation out of the parameter list (using
ParamExternData.textValue if set, calling the output function
otherwise), and stash that in the portal.
At the start of all the already existing PG_TRY blocks in pquery.c we
push an element onto the error_context_stack that adds the errdetail
with the parameters to the error. Alternatively we could also add them
in in the PG_CATCH blocks, but that'd only work for elevel == ERROR
(i.e. neither FATAL nor non throwing log levels would be able to enrich
the error).
Thinking about this: I think the current approach doesn't actually
handle recursive errors correctly. Even if we fail to emit the error
message due to the parameter details requiring a lot of memory, we'd
again do so (and again fail) when handling that OOM error, because
current_top_portal afaict would still be set. At the very least this'd
need to integrate with the recursion_depth logic in elog.c.
+ appendStringInfoCharMacro(¶m_str, '\''); + for (p = pstring; *p; p++) + { + if (*p == '\'') /* double single quotes */ + appendStringInfoCharMacro(¶m_str, *p); + appendStringInfoCharMacro(¶m_str, *p); + } + appendStringInfoCharMacro(¶m_str, '\'');I know this is old code, but: This is really inefficient. Will cause a
lot of unnecessary memory-reallocations for large text outputs, because
we don't immediately grow to at least close to the required size.Agreed, but we can't blame a patch because it moves around some old
crufty code. If Alexey wants to include another patch to change this to
a better formulation, I'm happy to push that before or after his main
patch. And if he doesn't want to, that's fine with me too.
Well, this patch makes it potentially a considerably hotter path, so I
think there's some justification for pushing a bit. But I'd not require
it either.
As I said, I think we cannot generate this string at error time, because
it makes it much much more likely to exhaust the error context - a bad
thing.
(Is doing a strlen first to enlarge the stringinfo an overall better
approach?)
Yes, it'd be better.
(I wonder if it would make sense to strchr each ' and memcpy the
intervening bytes ... if only that didn't break the SI abstraction
completely ...)
I'd probably just count the ' in one pass, enlarge the stringinfo to the
required size, and then put the string directly into he stringbuffer.
Possibly just putting the necessary code into stringinfo.c. We already
have multiple copies of this inefficient logic...
But even if not, I don't think writing data into the stringbuf directly
is that ugly, we do that in enough places that I'd argue that that's
basically part of how it's expected to be used.
In HEAD there's at least
- postgres.c:errdetail_params(),
- pl_exec.c:format_preparedparamsdata()
pl_exec.c:format_expr_params()
and
- dblink.c:escape_param_str()
- deparse.c:deparseStringLiteral()
- xlog.c:do_pg_start_backup() (after "Add the escape character"),
- tsearchcmds.c:serialize_deflist()
- ruleutils.c:simple_quote_literal()
are nearly the same.
Greetings,
Andres Freund
Hello Alvaro,
I didn't like abusing testlibpq3.c for your new stuff, so I moved it off
to a new file testlibpq5.c. I cleaned up a few other cosmetics things
about this -- v10 attached.
Thanks for doing this.
I eventually noticed that this patch fails
to initialize each param's textValue to NULL,
I've added the changes to set textValue to NULL for each parameter after
calling makeParamList.
I think it's the best option, as it's natural to populate text
representations (albeit with NULLs)
in the same loop as we write the other parameters attributes.
It still requires us to set hasTextValues afterwards, as it doesn't seem
practical to me to make
two loops, first null them all and next populate them: if there's a
parsing or converting problem
midway it's out of the feature scope and already being logged elsewhere.
Attaching v11, marking as waiting for review.
Best, Alex
Attachments:
log_parameters_v011.patchtext/x-patch; name=log_parameters_v011.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6612f95f9f..81dfee5fe7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6414,6 +6414,23 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether the statement is logged with bind parameter values.
+ It adds some overhead, as postgres will cache textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged. The default is
+ <literal>off</literal>. Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 7e0a041fab..7692ea551b 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -406,7 +406,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
prm->value = ExecEvalExprSwitchContext(n,
GetPerTupleExprContext(estate),
&prm->isnull);
-
+ prm->textValue = NULL;
i++;
}
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index 83337c2eed..92fcee8b19 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -925,6 +925,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
prm->isnull = fcinfo->args[i].isnull;
prm->pflags = 0;
prm->ptype = fcache->pinfo->argtypes[i];
+ prm->textValue = NULL;
}
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 2c0ae395ba..f6b4246b47 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2458,6 +2458,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
prm->isnull = (Nulls && Nulls[i] == 'n');
prm->pflags = PARAM_FLAG_CONST;
prm->ptype = argtypes[i];
+ prm->textValue = NULL;
}
}
else
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..bf8e3705af 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -91,6 +92,9 @@ copyParamList(ParamListInfo from)
continue;
get_typlenbyval(nprm->ptype, &typLen, &typByVal);
nprm->value = datumCopy(nprm->value, typByVal, typLen);
+
+ /* We don't copy text values, as no caller needs that at present. */
+ nprm->textValue = NULL;
}
return retval;
@@ -247,6 +251,9 @@ RestoreParamList(char **start_address)
/* Read datum/isnull. */
prm->value = datumRestore(start_address, &prm->isnull);
+
+ /* We don't copy text values, as no caller needs that at present. */
+ prm->textValue = NULL;
}
return paramLI;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e8d8e6f828..ec1e4092f5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1714,6 +1720,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1751,6 +1760,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1818,9 +1830,18 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * If we need the parameter string, keep it -- possibly making
+ * a copy first, if necessary. Otherwise get rid of it.
+ */
+ if (pstring)
+ {
+ if (need_text_values)
+ params->params[paramno].textValue =
+ pstring == pbuf.data ? pstrdup(pstring) : pstring;
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1846,6 +1867,23 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging.
+ *
+ * This could be optimized for certain built-in types, for
+ * which the output function is safe to call even in an
+ * aborted transaction.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1870,10 +1908,19 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /* Only now can we safely set this. */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1954,6 +2001,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2004,32 +2052,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2178,6 +2243,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2306,61 +2372,21 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ MemoryContext oldcontext;
+ char *params_message;
- pfree(param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ oldcontext = MemoryContextSwitchTo(MessageContext);
+ params_message = get_portal_bind_params(params);
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2678,6 +2704,86 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_params
+ * Get a string containing parameters data -- used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal or the portal is
+ * not valid, or the text representations of the parameters are not available.
+ * If returning a non-NULL value, it allocates memory for the returned string
+ * in the current context, and it's the caller's responsibility to pfree() it.
+ */
+char *
+get_portal_bind_params(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calculated, or
+ * call potentially user-defined I/O functions to convert the internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4031,10 +4137,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720ef3a..40896a2d09 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,19 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error &&
+ PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ appendCSVLiteral(&buf, params);
+ pfree(params);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2959,19 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), params);
+ pfree(params);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 2178e1cf5e..c034f88bfa 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -483,6 +483,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1293,6 +1294,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3a61..fccfff3a03 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -532,6 +532,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..316d3a2888 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index ec21f7e45c..ad6517414b 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -19,6 +19,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -27,6 +28,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -78,6 +81,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_params(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 6791e0cbc2..57acc28ef9 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
index e69de29bb2..69b65c512b 100644
--- a/src/test/examples/testlibpq5.c
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,116 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ /* divide by zero -- but server won't realize until execution */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero expected, got an error message from server: %s\n"
+ "Please make sure it has been logged with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
Hi Anders and Alvaro,
I must have missed this conversation branch when sending in v011.
Sorry about that.
I think it might be worthwhile to cross-reference
log_min_error_statement, as log_parameters_on_error will only take
effect when the
statement is logged due to log_min_error_statement.
Agreed, added some clarification.
I don't think "cache" is the right descriptor. Usually caching implies
trying to make repeated tasks faster, which isn't the case here.
Well, potentially it can, if we set log_min_error_statement to something
lower than error and emit tons of warnings. But it's not the primary use
case, so I just replaced it by the word "save".
What I'm suggesting is that PortalStart() would build a string
representation out of the parameter list (using
ParamExternData.textValue if set, calling the output function
otherwise), and stash that in the portal.At the start of all the already existing PG_TRY blocks in pquery.c we
push an element onto the error_context_stack that adds the errdetail
with the parameters to the error. Alternatively we could also add them
in in the PG_CATCH blocks, but that'd only work for elevel == ERROR
(i.e. neither FATAL nor non throwing log levels would be able to enrich
the error).
I'm a bit worried about going this way, as it makes us log
the query and its parameters too far apart in the code,
and it's trickier to make sure we never log parameters without the query.
I think logging the parameters should not be part of error_context_stack,
but rather a primary part of logging facility itself, like statement.
That's because whether we want to log parameters depends
on print_stmt in elog.c. With print_stmt being computed upon edata,
I'm not sure how to work it out nicely.
Thinking about this: I think the current approach doesn't actually
handle recursive errors correctly. Even if we fail to emit the error
message due to the parameter details requiring a lot of memory, we'd
again do so (and again fail) when handling that OOM error, because
current_top_portal afaict would still be set. At the very least this'd
need to integrate with the recursion_depth logic in elog.c.
We handle it the same way as we do it for debug_query_string itself:
if (in_error_recursion_trouble())
{
error_context_stack = NULL;
debug_query_string = NULL;
current_top_portal = NULL;
}
I'm attaching the patch with docs fixes.
Best regards,
Alexey
Attachments:
log_parameters_v013.patchtext/x-patch; name=log_parameters_v013.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0191ec84b1..832ebe57a3 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6492,6 +6492,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 7e0a041fab..7692ea551b 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -406,7 +406,7 @@ EvaluateParams(PreparedStatement *pstmt, List *params,
prm->value = ExecEvalExprSwitchContext(n,
GetPerTupleExprContext(estate),
&prm->isnull);
-
+ prm->textValue = NULL;
i++;
}
diff --git a/src/backend/executor/functions.c b/src/backend/executor/functions.c
index 83337c2eed..92fcee8b19 100644
--- a/src/backend/executor/functions.c
+++ b/src/backend/executor/functions.c
@@ -925,6 +925,7 @@ postquel_sub_params(SQLFunctionCachePtr fcache,
prm->isnull = fcinfo->args[i].isnull;
prm->pflags = 0;
prm->ptype = fcache->pinfo->argtypes[i];
+ prm->textValue = NULL;
}
}
else
diff --git a/src/backend/executor/spi.c b/src/backend/executor/spi.c
index 2c0ae395ba..f6b4246b47 100644
--- a/src/backend/executor/spi.c
+++ b/src/backend/executor/spi.c
@@ -2458,6 +2458,7 @@ _SPI_convert_params(int nargs, Oid *argtypes,
prm->isnull = (Nulls && Nulls[i] == 'n');
prm->pflags = PARAM_FLAG_CONST;
prm->ptype = argtypes[i];
+ prm->textValue = NULL;
}
}
else
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..bf8e3705af 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -45,6 +45,7 @@ makeParamList(int numParams)
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
retval->numParams = numParams;
+ retval->hasTextValues = false;
return retval;
}
@@ -91,6 +92,9 @@ copyParamList(ParamListInfo from)
continue;
get_typlenbyval(nprm->ptype, &typLen, &typByVal);
nprm->value = datumCopy(nprm->value, typByVal, typLen);
+
+ /* We don't copy text values, as no caller needs that at present. */
+ nprm->textValue = NULL;
}
return retval;
@@ -247,6 +251,9 @@ RestoreParamList(char **start_address)
/* Read datum/isnull. */
prm->value = datumRestore(start_address, &prm->isnull);
+
+ /* We don't copy text values, as no caller needs that at present. */
+ prm->textValue = NULL;
}
return paramLI;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 4bec40aa28..4b648a9f0a 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -86,6 +86,12 @@
*/
const char *debug_query_string; /* client-supplied query string */
+/*
+ * The top-level portal that the client is immediately working with:
+ * creating, binding, executing, or all at one using simple protocol
+ */
+Portal current_top_portal = NULL;
+
/* Note: whereToSendOutput is initialized for the bootstrap/standalone case */
CommandDest whereToSendOutput = DestDebug;
@@ -1716,6 +1722,9 @@ exec_bind_message(StringInfo input_message)
else
portal = CreatePortal(portal_name, false, false);
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
+
/*
* Prepare to copy stuff into the portal's memory context. We do all this
* copying first, because it could possibly fail (out-of-memory) and we
@@ -1753,6 +1762,9 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* GUC value can change, so we remember its state to be consistent */
+ bool need_text_values = log_parameters_on_error;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1820,9 +1832,18 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * If we need the parameter string, keep it -- possibly making
+ * a copy first, if necessary. Otherwise get rid of it.
+ */
+ if (pstring)
+ {
+ if (need_text_values)
+ params->params[paramno].textValue =
+ pstring == pbuf.data ? pstrdup(pstring) : pstring;
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1848,6 +1869,23 @@ exec_bind_message(StringInfo input_message)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("incorrect binary data format in bind parameter %d",
paramno + 1)));
+
+ /*
+ * Compute textual representation for further logging.
+ *
+ * This could be optimized for certain built-in types, for
+ * which the output function is safe to call even in an
+ * aborted transaction.
+ */
+ if (!isNull && need_text_values)
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(ptype, &typoutput, &typisvarlena);
+ params->params[paramno].textValue =
+ OidOutputFunctionCall(typoutput, pval);
+ }
}
else
{
@@ -1872,10 +1910,19 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /* Only now can we safely set this. */
+ params->hasTextValues = need_text_values;
}
else
params = NULL;
+ /*
+ * Set portal parameters early for them to get logged if an error happens
+ * on planning stage
+ */
+ portal->portalParams = params;
+
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
@@ -1956,6 +2003,7 @@ exec_bind_message(StringInfo input_message)
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2006,32 +2054,49 @@ exec_execute_message(const char *portal_name, long max_rows)
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
- /*
- * We must copy the sourceText and prepStmtName into MessageContext in
- * case the portal is destroyed during finish_xact_command. Can avoid the
- * copy if it's not an xact command, though.
- */
if (is_xact_command)
{
+ /*
+ * We must copy the sourceText and prepStmtName into MessageContext in
+ * case the portal is destroyed during finish_xact_command.
+ */
sourceText = pstrdup(portal->sourceText);
if (portal->prepStmtName)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
/*
* An xact command shouldn't have any parameters, which is a good
* thing because they wouldn't be around after finish_xact_command.
*/
portalParams = NULL;
+
+ /*
+ * We don't set current top portal for xact cmds, as
+ * 1) the portal may be destroyed earlier than the error happens
+ * 2) it anyway has no parameters in it to log
+ */
+ Assert(current_top_portal == NULL);
}
else
{
+ /*
+ * The portal will remain there until we leave the function, so it's
+ * safe avoid copying and just to refer to the data in its memory
+ * context instead
+ */
sourceText = portal->sourceText;
if (portal->prepStmtName)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
+
+ /*
+ * For non-xact commands, we remember the current portal and its bind
+ * parameters for further use in logging
+ */
+ Assert(current_top_portal == NULL);
+ current_top_portal = portal;
portalParams = portal->portalParams;
}
@@ -2183,6 +2248,7 @@ exec_execute_message(const char *portal_name, long max_rows)
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ current_top_portal = NULL;
debug_query_string = NULL;
}
@@ -2311,61 +2377,21 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
- {
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
+ MemoryContext oldcontext;
+ char *params_message;
- pfree(param_str.data);
+ /* Make sure any trash is generated in MessageContext */
+ oldcontext = MemoryContextSwitchTo(MessageContext);
+ params_message = get_portal_bind_params(params);
- MemoryContextSwitchTo(oldcontext);
+ if (params_message)
+ {
+ errdetail("parameters: %s", params_message);
+ pfree(params_message);
}
+ MemoryContextSwitchTo(oldcontext);
+
return 0;
}
@@ -2683,6 +2709,86 @@ drop_unnamed_stmt(void)
}
}
+/*
+ * get_portal_bind_params
+ * Get a string containing parameters data -- used for logging.
+ *
+ * Can return NULL if there are no parameters in the portal or the portal is
+ * not valid, or the text representations of the parameters are not available.
+ * If returning a non-NULL value, it allocates memory for the returned string
+ * in the current context, and it's the caller's responsibility to pfree() it.
+ */
+char *
+get_portal_bind_params(ParamListInfo params)
+{
+ StringInfoData param_str;
+
+ /* No parameters to format */
+ if (!params || params->numParams == 0)
+ return NULL;
+
+ /*
+ * We either need textual representation of parameters pre-calculated, or
+ * call potentially user-defined I/O functions to convert the internal
+ * representation into text, which cannot be done in an aborted xact
+ */
+ if (!params->hasTextValues && IsAbortedTransactionBlockState())
+ return NULL;
+
+ initStringInfo(¶m_str);
+
+ /* This code doesn't support dynamic param lists */
+ Assert(params->paramFetch == NULL);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *prm = ¶ms->params[paramno];
+ char *pstring;
+ char *p;
+
+ appendStringInfo(¶m_str, "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (prm->isnull)
+ {
+ appendStringInfoString(¶m_str, "NULL");
+ continue;
+ }
+
+ if (params->hasTextValues)
+ pstring = prm->textValue;
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ if (!OidIsValid(prm->ptype))
+ {
+ appendStringInfoString(¶m_str, "UNKNOWN TYPE");
+ continue;
+ }
+
+ getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, prm->value);
+ }
+
+ appendStringInfoCharMacro(¶m_str, '\'');
+ for (p = pstring; *p; p++)
+ {
+ if (*p == '\'') /* double single quotes */
+ appendStringInfoCharMacro(¶m_str, *p);
+ appendStringInfoCharMacro(¶m_str, *p);
+ }
+ appendStringInfoCharMacro(¶m_str, '\'');
+
+ if (!params->hasTextValues)
+ pfree(pstring);
+ }
+
+ return param_str.data;
+}
+
/* --------------------------------
* signal handler routines used in PostgresMain()
@@ -4035,10 +4141,11 @@ PostgresMain(int argc, char *argv[],
EmitErrorReport();
/*
- * Make sure debug_query_string gets reset before we possibly clobber
- * the storage it points at.
+ * Make sure these get reset before we possibly clobber
+ * the storages they point at.
*/
debug_query_string = NULL;
+ current_top_portal = NULL;
/*
* Abort the current transaction in order to recover.
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 8b4720ef3a..40896a2d09 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -76,6 +76,7 @@
#include "tcop/tcopprot.h"
#include "utils/guc.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
#include "utils/ps_status.h"
@@ -344,6 +345,7 @@ errstart(int elevel, const char *filename, int lineno,
{
error_context_stack = NULL;
debug_query_string = NULL;
+ current_top_portal = NULL;
}
}
if (++errordata_stack_depth >= ERRORDATA_STACK_SIZE)
@@ -2788,6 +2790,19 @@ write_csvlog(ErrorData *edata)
if (print_stmt && edata->cursorpos > 0)
appendStringInfo(&buf, "%d", edata->cursorpos);
appendStringInfoChar(&buf, ',');
+ if (print_stmt && log_parameters_on_error &&
+ PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ appendCSVLiteral(&buf, params);
+ pfree(params);
+ }
+ }
+ appendStringInfoChar(&buf, ',');
/* file error location */
if (Log_error_verbosity >= PGERROR_VERBOSE)
@@ -2944,6 +2959,19 @@ send_message_to_server_log(ErrorData *edata)
appendStringInfoString(&buf, _("STATEMENT: "));
append_with_tabs(&buf, debug_query_string);
appendStringInfoChar(&buf, '\n');
+
+ if (log_parameters_on_error && PortalIsValid(current_top_portal))
+ {
+ char *params;
+
+ params = get_portal_bind_params(current_top_portal->portalParams);
+ if (params != NULL)
+ {
+ log_line_prefix(&buf, edata);
+ appendStringInfo(&buf, _("PARAMETERS: %s\n"), params);
+ pfree(params);
+ }
+ }
}
#ifdef HAVE_SYSLOG
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 31a5ef0474..9ccd2c2bf0 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -483,6 +483,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1293,6 +1294,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3a61..fccfff3a03 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -532,6 +532,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..316d3a2888 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -93,6 +93,7 @@ typedef struct ParamExternData
bool isnull; /* is it NULL? */
uint16 pflags; /* flag bits, see above */
Oid ptype; /* parameter's datatype, or 0 */
+ char *textValue; /* textual representation for debug purposes */
} ParamExternData;
typedef struct ParamListInfoData *ParamListInfo;
@@ -116,6 +117,8 @@ typedef struct ParamListInfoData
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
int numParams; /* nominal/maximum # of Params represented */
+ bool hasTextValues; /* whether textValue for all non-null
+ params is populated */
/*
* params[] may be of length zero if paramFetch is supplied; otherwise it
diff --git a/src/include/tcop/tcopprot.h b/src/include/tcop/tcopprot.h
index ec21f7e45c..ad6517414b 100644
--- a/src/include/tcop/tcopprot.h
+++ b/src/include/tcop/tcopprot.h
@@ -19,6 +19,7 @@
#include "nodes/plannodes.h"
#include "storage/procsignal.h"
#include "utils/guc.h"
+#include "utils/portal.h"
#include "utils/queryenvironment.h"
@@ -27,6 +28,8 @@
extern CommandDest whereToSendOutput;
extern PGDLLIMPORT const char *debug_query_string;
+extern PGDLLIMPORT Portal current_top_portal;
+
extern int max_stack_depth;
extern int PostAuthDelay;
@@ -78,6 +81,7 @@ extern long get_stack_depth_rlimit(void);
extern void ResetUsage(void);
extern void ShowUsage(const char *title);
extern int check_log_duration(char *msec_str, bool was_logged);
+extern char *get_portal_bind_params(ParamListInfo params);
extern void set_debug_options(int debug_flag,
GucContext context, GucSource source);
extern bool set_plan_disabling_options(const char *arg,
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 6791e0cbc2..57acc28ef9 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..69b65c512b
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,116 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ /* divide by zero -- but server won't realize until execution */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero expected, got an error message from server: %s\n"
+ "Please make sure it has been logged with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
So, if some parameters are large (they can be up to 1 GB-1, remember)
then we can bloat the log file severely. I think we need to place an
upper limit on the strings that we're going to log -- as inspiration,
callers of ExecBuildValueDescription uses 64 chars per value maximum.
Something like that seems reasonable. So I think you need to add some
pg_mbcliplen() calls in a couple of places in exec_bind_message.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-Nov-07, Alvaro Herrera wrote:
So, if some parameters are large (they can be up to 1 GB-1, remember)
then we can bloat the log file severely. I think we need to place an
upper limit on the strings that we're going to log -- as inspiration,
callers of ExecBuildValueDescription uses 64 chars per value maximum.
Something like that seems reasonable. So I think you need to add some
pg_mbcliplen() calls in a couple of places in exec_bind_message.
(BTW it looks like plpgsql_param_fetch() is creating ParamExternData
values and not setting textValue for them.)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi,
On 2019-11-05 12:07:50 +0000, Alexey Bashtanov wrote:
What I'm suggesting is that PortalStart() would build a string
representation out of the parameter list (using
ParamExternData.textValue if set, calling the output function
otherwise), and stash that in the portal.At the start of all the already existing PG_TRY blocks in pquery.c we
push an element onto the error_context_stack that adds the errdetail
with the parameters to the error. Alternatively we could also add them
in in the PG_CATCH blocks, but that'd only work for elevel == ERROR
(i.e. neither FATAL nor non throwing log levels would be able to enrich
the error).I'm a bit worried about going this way, as it makes us log
the query and its parameters too far apart in the code,
and it's trickier to make sure we never log parameters without the query.
The way you do it you need to do it in numerous places, and I'm pretty
sure you're missing some already. E.g. this will not work to log
parameters for parametrized statements generated on the server side,
e.g. for foreign key queries. I don't think that's the right direction
to go. You can maybe argue that we don't need support for logging server
side generated parameters in the initial version, but the approach
should be compatible with adding that support.
I think logging the parameters should not be part of error_context_stack,
but rather a primary part of logging facility itself, like statement.
That's because whether we want to log parameters depends
on print_stmt in elog.c. With print_stmt being computed upon edata,
I'm not sure how to work it out nicely.
I don't think those really are contradictions. You can continue to have
an errdetail_params(), and but call it from the error context callback
set up in the portal code.
That'd also get rid of the two different copies of the logic for
deciding whether to log bind parameters. Right now your approach
e.g. doesn't work for emit_log_hook. Nor is all that clear to me that
we'd never want to send this information to the client.
Even leaving that aside, I'm *STRONGLY* against entangling elog.c with
query execution details like ParamListInfo. We should work to make
elog.c know less about different parts of the system, not more.
Thinking about this: I think the current approach doesn't actually
handle recursive errors correctly. Even if we fail to emit the error
message due to the parameter details requiring a lot of memory, we'd
again do so (and again fail) when handling that OOM error, because
current_top_portal afaict would still be set. At the very least this'd
need to integrate with the recursion_depth logic in elog.c.
We handle it the same way as we do it for debug_query_string itself:
��� ��� if (in_error_recursion_trouble())
��� ��� {
��� ��� ��� error_context_stack = NULL;
��� ��� ��� debug_query_string = NULL;
��� ��� ��� current_top_portal = NULL;
��� ��� }
Sorry, had missed that, it wasn't apparent in the diff. I personally
just always use a larger context size to make it easier to recognize...
Greetings,
Andres Freund
On Thu, Nov 07, 2019 at 03:41:04PM -0800, Andres Freund wrote:
The way you do it you need to do it in numerous places, and I'm pretty
sure you're missing some already. E.g. this will not work to log
parameters for parametrized statements generated on the server side,
e.g. for foreign key queries. I don't think that's the right direction
to go. You can maybe argue that we don't need support for logging server
side generated parameters in the initial version, but the approach
should be compatible with adding that support.
This patch had a review from two committers, with no updates from the
author, so marked as returned with feedback.
--
Michael
Hi,
I'm sorry for replying so late.
I don't think those really are contradictions. You can continue to have
an errdetail_params(), and but call it from the error context callback
set up in the portal code
...
Even leaving that aside, I'm *STRONGLY* against entangling elog.c with
query execution details like ParamListInfo. We should work to make
elog.c know less about different parts of the system, not more.
I've implemented it using error contexts, please could you have a look
at the patch attached?
I did not use the PG_TRY blocks in portal code because they are not wide
enough.
Otherwise, we wouldn't catch errors that can happen in GetCachedPlan.
Instead I added an item to error context chain in postgres.c
exec_*_message routines.
I can't find why it could be unsafe, but I'm still a bit worried about
it as there's no
other error context changes that early in the call stack.
I've also made string formatting more efficient, and pre-calculate
the whole string instead of individual values as suggested. Unfortunately
it still copies the error text when reporting -- I'm not sure how I
change it
without substantial change of error logging infrastructure.
One more concern I'd like to ask for opinions. I'm using errdetail_log
for parameters logging, as it's the only way to make it appear in the
log only
but not get sent to the client. It looks a bit awkward, as it appears like
ERROR
DETAIL
CONTEXT
STATEMENT,
where CONTEXT may in fact be something inner nested than the parameters
that appear in DETAILS.
With this regards, I'm thinking of adding some special arrangements into
elog.c.
One idea is a econtext_log to add a log-only data into CONTEXT, however
it'll still log
params above the statement, although directly above.
Another option is a special estatementparams (or perhaps
efinalcontext_log?) to log
below the statement.
What do you think?
2 Alvaro:
So, if some parameters are large (they can be up to 1 GB-1, remember)
then we can bloat the log file severely. I think we need to place an
upper limit on the strings that we're going to log -- as inspiration,
callers of ExecBuildValueDescription uses 64 chars per value maximum.
Something like that seems reasonable. So I think you need to add some
pg_mbcliplen() calls in a couple of places in exec_bind_message.
I like the idea, but I don't think it's directly related to the change
proposed. We are already logging parameters in certain situations
with no limits on their lenghts. Given the time it takes to get the change
through (not least because of me addressing reviewers' comments
really slowly) I'd not include it in the patch. Do you find it acceptable?
Best, Alex
Attachments:
log_parameters_v012.patchtext/x-patch; name=log_parameters_v012.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4ec13f3311..b3a0d27861 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6597,6 +6597,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..7c1f86f5c5 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,12 +15,16 @@
#include "postgres.h"
+#include "lib/stringinfo.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
+#include "utils/guc.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+static void LogParams(void *arg);
/*
* Allocate and initialize a new ParamListInfo structure.
@@ -44,6 +48,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->logString = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +63,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We also don't copy logString.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -221,6 +228,8 @@ SerializeParamList(ParamListInfo paramLI, char **start_address)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We also don't copy logString.
*/
ParamListInfo
RestoreParamList(char **start_address)
@@ -251,3 +260,152 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString - create a string to represent parameters list for
+ * logging and save it to params. If caller already knows textual
+ * representations for some or all parameters, it can pass an array of exactly
+ * params->numParams values as knownTextValues. It can contain NULLs for the
+ * individual values not known, or, if no text text values known at all the
+ * caller may pass NULL pointer.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ int logStringLen = 0;
+ StringInfoData logString;
+ char **textValues;
+
+ if (params->logString != NULL)
+ return;
+
+ Assert(params->paramFetch == NULL);
+
+ initStringInfo(&logString);
+
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ textValues = (char **) palloc0(params->numParams * sizeof(char *));
+
+ /* calculate unknown text representations and pre-calculate byte lengths */
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData param = params->params[paramno];
+
+ /* reserve the space for the number of digits in paramno + 1 */
+ for (int d = paramno + 1; d > 0; d /= 10)
+ logStringLen++;
+
+ if (param.isnull)
+ {
+ logStringLen += 4; /* for "NULL" */
+ }
+ else
+ {
+ char *textValue,
+ *s;
+
+ /* find out textValues[paramno] */
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ {
+ textValues[paramno] = knownTextValues[paramno];
+ }
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ getTypeOutputInfo(param.ptype, &typoutput, &typisvarlena);
+ textValues[paramno] =
+ OidOutputFunctionCall(typoutput, param.value);
+ }
+
+ /* caluclate the space needed its printed length */
+ textValue = textValues[paramno];
+ for (s = textValue; *s != '\0'; s++)
+ if (*s == '\'')
+ logStringLen++; /* for quotes doubling */
+ /* string length + 2 for quotes */
+ logStringLen += s - textValue + 2;
+ }
+ }
+
+ /*
+ * "$1 = something, " -- 6 bytes extra to the ordinal and the value;
+ * ^ ^^^ ^^
+ * for the last one we don't need the comma and the space but need 0 byte
+ */
+ logStringLen += params->numParams * 6 - 1;
+
+ MemoryContextSwitchTo(oldCxt);
+
+ /* enlarge at once to avoid multiple reallocations */
+ enlargeStringInfo(&logString, logStringLen);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ appendStringInfo(&logString,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+ appendStringInfoStringQuoted(&logString, textValues[paramno]);
+ }
+
+ params->logString = logString.data;
+
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * PutParamsInErrorContext - if needed and possible,
+ * add an error context entry with bind parameters
+ *
+ * params is a pointer to ParamListInfo for the caller to be able to clear it
+ * if the actual ParamListInfo is gone
+ */
+void
+PutParamsInErrorContext(ErrorContextCallback *new_error_context_entry,
+ ParamListInfo *params)
+{
+ /*
+ * Setting this anyway for the caller to be able to revert this operation
+ * with `error_context_stack = new_error_context_entry->previous` regardless
+ * whether we actually change the stack
+ */
+ new_error_context_entry->previous = error_context_stack;
+
+ if (!log_parameters_on_error || *params == NULL
+ || (*params)->logString == NULL)
+ return;
+
+ new_error_context_entry->callback = LogParams;
+ new_error_context_entry->arg = (void *)(params);
+ error_context_stack = new_error_context_entry;
+}
+
+/*
+ * LogParams - callback for printing parameters in error context
+ */
+static void
+LogParams(void *arg)
+{
+ ParamListInfo params = *((ParamListInfo *)(arg));
+
+ /*
+ * XXX: errdetail_log copies the error message formatting, so we're
+ * allocating a potentially large amounts of memory for the whole parameter
+ * string again
+ *
+ * We can have NULL in params->logString if log_parameters_on_error was off
+ * when we decided whether to call BuildParamLogString. In this rare case
+ * we don't log the parameters: building the log string now wouldn't be safe
+ * as we must't call user-defined I/O functions when in an aborted xact.
+ */
+ if (params != NULL && params->logString != NULL
+ && log_parameters_on_error && geterrshowstmt())
+ errdetail_log("parameters: %s", params->logString);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..9b562f7f01 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -177,7 +177,7 @@ static void forbidden_in_wal_sender(char firstchar);
static List *pg_rewrite_query(Query *query);
static bool check_log_statement(List *stmt_list);
static int errdetail_execute(List *raw_parsetree_list);
-static int errdetail_params(ParamListInfo params);
+static int errdetail_portal_params(Portal portal);
static int errdetail_abort(void);
static int errdetail_recovery_conflict(void);
static void start_xact_command(void);
@@ -1596,23 +1596,24 @@ exec_parse_message(const char *query_string, /* string to execute */
static void
exec_bind_message(StringInfo input_message)
{
- const char *portal_name;
- const char *stmt_name;
- int numPFormats;
- int16 *pformats = NULL;
- int numParams;
- int numRFormats;
- int16 *rformats = NULL;
- CachedPlanSource *psrc;
- CachedPlan *cplan;
- Portal portal;
- char *query_string;
- char *saved_stmt_name;
- ParamListInfo params;
- MemoryContext oldContext;
- bool save_log_statement_stats = log_statement_stats;
- bool snapshot_set = false;
- char msec_str[32];
+ const char *portal_name;
+ const char *stmt_name;
+ int numPFormats;
+ int16 *pformats = NULL;
+ int numParams;
+ int numRFormats;
+ int16 *rformats = NULL;
+ CachedPlanSource *psrc;
+ CachedPlan *cplan;
+ Portal portal;
+ char *query_string;
+ char *saved_stmt_name;
+ ParamListInfo params;
+ MemoryContext oldContext;
+ bool save_log_statement_stats = log_statement_stats;
+ bool snapshot_set = false;
+ char msec_str[32];
+ ErrorContextCallback params_error_context;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1752,6 +1753,10 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* These are short-living, so we don't pollute the portal context */
+ char **knownTextValues = MemoryContextAllocZero(MessageContext,
+ numParams * sizeof(char *));
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1819,9 +1824,17 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * If we need the parameter string, save it.
+ * If not -- get rid of it.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ knownTextValues[paramno] = pstring;
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1871,6 +1884,9 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues);
}
else
params = NULL;
@@ -1878,6 +1894,9 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* From now on, log parameters on error if enabled in GUC */
+ PutParamsInErrorContext(¶ms_error_context, ¶ms);
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1948,13 +1967,14 @@ exec_bind_message(StringInfo input_message)
*portal_name ? portal_name : "",
psrc->query_string),
errhidestmt(true),
- errdetail_params(params)));
+ errdetail_portal_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ error_context_stack = params_error_context.previous;
debug_query_string = NULL;
}
@@ -1966,19 +1986,20 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
- DestReceiver *receiver;
- Portal portal;
- bool completed;
- char completionTag[COMPLETION_TAG_BUFSIZE];
- const char *sourceText;
- const char *prepStmtName;
- ParamListInfo portalParams;
- bool save_log_statement_stats = log_statement_stats;
- bool is_xact_command;
- bool execute_is_fetch;
- bool was_logged = false;
- char msec_str[32];
+ CommandDest dest;
+ DestReceiver *receiver;
+ Portal portal;
+ ParamListInfo params;
+ bool completed;
+ char completionTag[COMPLETION_TAG_BUFSIZE];
+ const char *sourceText;
+ const char *prepStmtName;
+ bool save_log_statement_stats = log_statement_stats;
+ bool is_xact_command;
+ bool execute_is_fetch;
+ bool was_logged = false;
+ char msec_str[32];
+ ErrorContextCallback params_error_context;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2002,6 +2023,10 @@ exec_execute_message(const char *portal_name, long max_rows)
return;
}
+ /* From now on, log parameters on error if enabled in GUC */
+ params = portal->portalParams;
+ PutParamsInErrorContext(¶ms_error_context, ¶ms);
+
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
@@ -2017,12 +2042,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
@@ -2031,7 +2050,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2083,7 +2101,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_portal_params(portal)));
was_logged = true;
}
@@ -2122,6 +2140,15 @@ exec_execute_message(const char *portal_name, long max_rows)
{
if (is_xact_command)
{
+ /*
+ * The portal and its params may be destroyed during
+ * finish_xact_command below. We only may need the portal for
+ * parameters logging, but an xact command shouldn't have any
+ * parameters.
+ */
+ portal = NULL;
+ params = NULL;
+
/*
* If this was a transaction control statement, commit it. We
* will start a new xact command for the next command (if any).
@@ -2175,13 +2202,14 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_portal_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ error_context_stack = params_error_context.previous;
debug_query_string = NULL;
}
@@ -2321,66 +2349,26 @@ errdetail_execute(List *raw_parsetree_list)
}
/*
- * errdetail_params
+ * errdetail_portal_params
*
* Add an errdetail() line showing bind-parameter data, if available.
*/
static int
-errdetail_params(ParamListInfo params)
+errdetail_portal_params(Portal portal)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (portal != NULL &&
+ portal->portalParams &&
+ portal->portalParams->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
+ /*
+ * We switch back to portal context for the params logString to be
+ * generated there in order to reuse it in later invocations
+ */
+ MemoryContext old_cxt = MemoryContextSwitchTo(portal->portalContext);
+ BuildParamLogString(portal->portalParams, NULL);
+ MemoryContextSwitchTo(old_cxt);
- MemoryContextSwitchTo(oldcontext);
+ errdetail("parameters: %s", portal->portalParams->logString);
}
return 0;
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index f46653632e..eb39c750f4 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -1363,6 +1363,22 @@ getinternalerrposition(void)
return edata->internalpos;
}
+/*
+ * geterrshowstmt --- return whether we are going to print the statement in the log
+ *
+ * This is only intended for use in error callback subroutines, since there
+ * is no other place outside elog.c where the concept is meaningful.
+ */
+bool
+geterrshowstmt(void)
+{
+ ErrorData *edata = &errordata[errordata_stack_depth];
+
+ return is_log_level_output(edata->elevel, log_min_error_statement) &&
+ debug_query_string != NULL &&
+ !edata->hide_stmt;
+}
+
/*
* elog_start --- startup for old-style API
@@ -2742,7 +2758,7 @@ static void
write_csvlog(ErrorData *edata)
{
StringInfoData buf;
- bool print_stmt = false;
+ bool print_stmt = geterrshowstmt();
/* static counter for line numbers */
static long log_line_number = 0;
@@ -2886,10 +2902,6 @@ write_csvlog(ErrorData *edata)
appendStringInfoChar(&buf, ',');
/* user query --- only reported if not disabled by the caller */
- if (is_log_level_output(edata->elevel, log_min_error_statement) &&
- debug_query_string != NULL &&
- !edata->hide_stmt)
- print_stmt = true;
if (print_stmt)
appendCSVLiteral(&buf, debug_query_string);
appendStringInfoChar(&buf, ',');
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5fccc9683e..1be1f1fb29 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 46a06ffacd..91cdc06fc4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..300fa4527b 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,42 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well*/
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk */
+ appendBinaryStringInfoNT(str, chunk_copy_start, strlen(chunk_copy_start));
+
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..4978ac9aa3 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..49a450b6d0 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *logString; /* string to log parameters */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -156,5 +157,9 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues);
+extern void PutParamsInErrorContext(
+ ErrorContextCallback *new_error_context_entry,
+ ParamListInfo *params);
#endif /* PARAMS_H */
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index 9ab4b5470b..ecc78d2ae6 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -202,6 +202,7 @@ extern int err_generic_string(int field, const char *str);
extern int geterrcode(void);
extern int geterrposition(void);
extern int getinternalerrposition(void);
+extern bool geterrshowstmt(void);
/*----------
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..47a5e31335
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,193 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /* emit error on parse stage */
+ paramValues[0] = (char *) "non-number -- parse";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT $1::int",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("text->int conversion error expected on parse stage, "
+ "got an error message from server: %s\n",
+ PQerrorMessage(conn));
+
+ /* emit error on bind stage */
+ paramValues[0] = (char *) "{malformed json -- bind :-{";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT (' ' || $1::text || ' ')::json",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Json parsing error expected on bind stage, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 0 params */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int /*exec*/",
+ 0, /* # params */
+ NULL, /* let the backend deduce param type */
+ NULL,
+ NULL,
+ NULL,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 3 params */
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ res = PQexecParams(
+ conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text /*exec*/",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1 /* ask for binary results */
+ );
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
Patch file name is not great, changing it.
Show quoted text
On 30/11/2019 12:35, Alexey Bashtanov wrote:
I've implemented it using error contexts, please could you have a look
at the patch attached?
Attachments:
log_parameters_v014.patchtext/x-patch; name=log_parameters_v014.patchDownload
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4ec13f3311..b3a0d27861 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6597,6 +6597,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..7c1f86f5c5 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,12 +15,16 @@
#include "postgres.h"
+#include "lib/stringinfo.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
+#include "utils/guc.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+static void LogParams(void *arg);
/*
* Allocate and initialize a new ParamListInfo structure.
@@ -44,6 +48,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->logString = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +63,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We also don't copy logString.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -221,6 +228,8 @@ SerializeParamList(ParamListInfo paramLI, char **start_address)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * We also don't copy logString.
*/
ParamListInfo
RestoreParamList(char **start_address)
@@ -251,3 +260,152 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString - create a string to represent parameters list for
+ * logging and save it to params. If caller already knows textual
+ * representations for some or all parameters, it can pass an array of exactly
+ * params->numParams values as knownTextValues. It can contain NULLs for the
+ * individual values not known, or, if no text text values known at all the
+ * caller may pass NULL pointer.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ int logStringLen = 0;
+ StringInfoData logString;
+ char **textValues;
+
+ if (params->logString != NULL)
+ return;
+
+ Assert(params->paramFetch == NULL);
+
+ initStringInfo(&logString);
+
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ textValues = (char **) palloc0(params->numParams * sizeof(char *));
+
+ /* calculate unknown text representations and pre-calculate byte lengths */
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData param = params->params[paramno];
+
+ /* reserve the space for the number of digits in paramno + 1 */
+ for (int d = paramno + 1; d > 0; d /= 10)
+ logStringLen++;
+
+ if (param.isnull)
+ {
+ logStringLen += 4; /* for "NULL" */
+ }
+ else
+ {
+ char *textValue,
+ *s;
+
+ /* find out textValues[paramno] */
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ {
+ textValues[paramno] = knownTextValues[paramno];
+ }
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ getTypeOutputInfo(param.ptype, &typoutput, &typisvarlena);
+ textValues[paramno] =
+ OidOutputFunctionCall(typoutput, param.value);
+ }
+
+ /* caluclate the space needed its printed length */
+ textValue = textValues[paramno];
+ for (s = textValue; *s != '\0'; s++)
+ if (*s == '\'')
+ logStringLen++; /* for quotes doubling */
+ /* string length + 2 for quotes */
+ logStringLen += s - textValue + 2;
+ }
+ }
+
+ /*
+ * "$1 = something, " -- 6 bytes extra to the ordinal and the value;
+ * ^ ^^^ ^^
+ * for the last one we don't need the comma and the space but need 0 byte
+ */
+ logStringLen += params->numParams * 6 - 1;
+
+ MemoryContextSwitchTo(oldCxt);
+
+ /* enlarge at once to avoid multiple reallocations */
+ enlargeStringInfo(&logString, logStringLen);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ appendStringInfo(&logString,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+ appendStringInfoStringQuoted(&logString, textValues[paramno]);
+ }
+
+ params->logString = logString.data;
+
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * PutParamsInErrorContext - if needed and possible,
+ * add an error context entry with bind parameters
+ *
+ * params is a pointer to ParamListInfo for the caller to be able to clear it
+ * if the actual ParamListInfo is gone
+ */
+void
+PutParamsInErrorContext(ErrorContextCallback *new_error_context_entry,
+ ParamListInfo *params)
+{
+ /*
+ * Setting this anyway for the caller to be able to revert this operation
+ * with `error_context_stack = new_error_context_entry->previous` regardless
+ * whether we actually change the stack
+ */
+ new_error_context_entry->previous = error_context_stack;
+
+ if (!log_parameters_on_error || *params == NULL
+ || (*params)->logString == NULL)
+ return;
+
+ new_error_context_entry->callback = LogParams;
+ new_error_context_entry->arg = (void *)(params);
+ error_context_stack = new_error_context_entry;
+}
+
+/*
+ * LogParams - callback for printing parameters in error context
+ */
+static void
+LogParams(void *arg)
+{
+ ParamListInfo params = *((ParamListInfo *)(arg));
+
+ /*
+ * XXX: errdetail_log copies the error message formatting, so we're
+ * allocating a potentially large amounts of memory for the whole parameter
+ * string again
+ *
+ * We can have NULL in params->logString if log_parameters_on_error was off
+ * when we decided whether to call BuildParamLogString. In this rare case
+ * we don't log the parameters: building the log string now wouldn't be safe
+ * as we must't call user-defined I/O functions when in an aborted xact.
+ */
+ if (params != NULL && params->logString != NULL
+ && log_parameters_on_error && geterrshowstmt())
+ errdetail_log("parameters: %s", params->logString);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..9b562f7f01 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -177,7 +177,7 @@ static void forbidden_in_wal_sender(char firstchar);
static List *pg_rewrite_query(Query *query);
static bool check_log_statement(List *stmt_list);
static int errdetail_execute(List *raw_parsetree_list);
-static int errdetail_params(ParamListInfo params);
+static int errdetail_portal_params(Portal portal);
static int errdetail_abort(void);
static int errdetail_recovery_conflict(void);
static void start_xact_command(void);
@@ -1596,23 +1596,24 @@ exec_parse_message(const char *query_string, /* string to execute */
static void
exec_bind_message(StringInfo input_message)
{
- const char *portal_name;
- const char *stmt_name;
- int numPFormats;
- int16 *pformats = NULL;
- int numParams;
- int numRFormats;
- int16 *rformats = NULL;
- CachedPlanSource *psrc;
- CachedPlan *cplan;
- Portal portal;
- char *query_string;
- char *saved_stmt_name;
- ParamListInfo params;
- MemoryContext oldContext;
- bool save_log_statement_stats = log_statement_stats;
- bool snapshot_set = false;
- char msec_str[32];
+ const char *portal_name;
+ const char *stmt_name;
+ int numPFormats;
+ int16 *pformats = NULL;
+ int numParams;
+ int numRFormats;
+ int16 *rformats = NULL;
+ CachedPlanSource *psrc;
+ CachedPlan *cplan;
+ Portal portal;
+ char *query_string;
+ char *saved_stmt_name;
+ ParamListInfo params;
+ MemoryContext oldContext;
+ bool save_log_statement_stats = log_statement_stats;
+ bool snapshot_set = false;
+ char msec_str[32];
+ ErrorContextCallback params_error_context;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1752,6 +1753,10 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ /* These are short-living, so we don't pollute the portal context */
+ char **knownTextValues = MemoryContextAllocZero(MessageContext,
+ numParams * sizeof(char *));
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1819,9 +1824,17 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * If we need the parameter string, save it.
+ * If not -- get rid of it.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ knownTextValues[paramno] = pstring;
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1871,6 +1884,9 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues);
}
else
params = NULL;
@@ -1878,6 +1894,9 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* From now on, log parameters on error if enabled in GUC */
+ PutParamsInErrorContext(¶ms_error_context, ¶ms);
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1948,13 +1967,14 @@ exec_bind_message(StringInfo input_message)
*portal_name ? portal_name : "",
psrc->query_string),
errhidestmt(true),
- errdetail_params(params)));
+ errdetail_portal_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("BIND MESSAGE STATISTICS");
+ error_context_stack = params_error_context.previous;
debug_query_string = NULL;
}
@@ -1966,19 +1986,20 @@ exec_bind_message(StringInfo input_message)
static void
exec_execute_message(const char *portal_name, long max_rows)
{
- CommandDest dest;
- DestReceiver *receiver;
- Portal portal;
- bool completed;
- char completionTag[COMPLETION_TAG_BUFSIZE];
- const char *sourceText;
- const char *prepStmtName;
- ParamListInfo portalParams;
- bool save_log_statement_stats = log_statement_stats;
- bool is_xact_command;
- bool execute_is_fetch;
- bool was_logged = false;
- char msec_str[32];
+ CommandDest dest;
+ DestReceiver *receiver;
+ Portal portal;
+ ParamListInfo params;
+ bool completed;
+ char completionTag[COMPLETION_TAG_BUFSIZE];
+ const char *sourceText;
+ const char *prepStmtName;
+ bool save_log_statement_stats = log_statement_stats;
+ bool is_xact_command;
+ bool execute_is_fetch;
+ bool was_logged = false;
+ char msec_str[32];
+ ErrorContextCallback params_error_context;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2002,6 +2023,10 @@ exec_execute_message(const char *portal_name, long max_rows)
return;
}
+ /* From now on, log parameters on error if enabled in GUC */
+ params = portal->portalParams;
+ PutParamsInErrorContext(¶ms_error_context, ¶ms);
+
/* Does the portal contain a transaction command? */
is_xact_command = IsTransactionStmtList(portal->stmts);
@@ -2017,12 +2042,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = pstrdup(portal->prepStmtName);
else
prepStmtName = "<unnamed>";
-
- /*
- * An xact command shouldn't have any parameters, which is a good
- * thing because they wouldn't be around after finish_xact_command.
- */
- portalParams = NULL;
}
else
{
@@ -2031,7 +2050,6 @@ exec_execute_message(const char *portal_name, long max_rows)
prepStmtName = portal->prepStmtName;
else
prepStmtName = "<unnamed>";
- portalParams = portal->portalParams;
}
/*
@@ -2083,7 +2101,7 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_portal_params(portal)));
was_logged = true;
}
@@ -2122,6 +2140,15 @@ exec_execute_message(const char *portal_name, long max_rows)
{
if (is_xact_command)
{
+ /*
+ * The portal and its params may be destroyed during
+ * finish_xact_command below. We only may need the portal for
+ * parameters logging, but an xact command shouldn't have any
+ * parameters.
+ */
+ portal = NULL;
+ params = NULL;
+
/*
* If this was a transaction control statement, commit it. We
* will start a new xact command for the next command (if any).
@@ -2175,13 +2202,14 @@ exec_execute_message(const char *portal_name, long max_rows)
*portal_name ? portal_name : "",
sourceText),
errhidestmt(true),
- errdetail_params(portalParams)));
+ errdetail_portal_params(portal)));
break;
}
if (save_log_statement_stats)
ShowUsage("EXECUTE MESSAGE STATISTICS");
+ error_context_stack = params_error_context.previous;
debug_query_string = NULL;
}
@@ -2321,66 +2349,26 @@ errdetail_execute(List *raw_parsetree_list)
}
/*
- * errdetail_params
+ * errdetail_portal_params
*
* Add an errdetail() line showing bind-parameter data, if available.
*/
static int
-errdetail_params(ParamListInfo params)
+errdetail_portal_params(Portal portal)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (portal != NULL &&
+ portal->portalParams &&
+ portal->portalParams->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
-
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
- char *p;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
+ /*
+ * We switch back to portal context for the params logString to be
+ * generated there in order to reuse it in later invocations
+ */
+ MemoryContext old_cxt = MemoryContextSwitchTo(portal->portalContext);
+ BuildParamLogString(portal->portalParams, NULL);
+ MemoryContextSwitchTo(old_cxt);
- MemoryContextSwitchTo(oldcontext);
+ errdetail("parameters: %s", portal->portalParams->logString);
}
return 0;
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index f46653632e..eb39c750f4 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -1363,6 +1363,22 @@ getinternalerrposition(void)
return edata->internalpos;
}
+/*
+ * geterrshowstmt --- return whether we are going to print the statement in the log
+ *
+ * This is only intended for use in error callback subroutines, since there
+ * is no other place outside elog.c where the concept is meaningful.
+ */
+bool
+geterrshowstmt(void)
+{
+ ErrorData *edata = &errordata[errordata_stack_depth];
+
+ return is_log_level_output(edata->elevel, log_min_error_statement) &&
+ debug_query_string != NULL &&
+ !edata->hide_stmt;
+}
+
/*
* elog_start --- startup for old-style API
@@ -2742,7 +2758,7 @@ static void
write_csvlog(ErrorData *edata)
{
StringInfoData buf;
- bool print_stmt = false;
+ bool print_stmt = geterrshowstmt();
/* static counter for line numbers */
static long log_line_number = 0;
@@ -2886,10 +2902,6 @@ write_csvlog(ErrorData *edata)
appendStringInfoChar(&buf, ',');
/* user query --- only reported if not disabled by the caller */
- if (is_log_level_output(edata->elevel, log_min_error_statement) &&
- debug_query_string != NULL &&
- !edata->hide_stmt)
- print_stmt = true;
if (print_stmt)
appendCSVLiteral(&buf, debug_query_string);
appendStringInfoChar(&buf, ',');
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 5fccc9683e..1be1f1fb29 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 46a06ffacd..91cdc06fc4 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..300fa4527b 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,42 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well*/
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk */
+ appendBinaryStringInfoNT(str, chunk_copy_start, strlen(chunk_copy_start));
+
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..4978ac9aa3 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..49a450b6d0 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *logString; /* string to log parameters */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -156,5 +157,9 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues);
+extern void PutParamsInErrorContext(
+ ErrorContextCallback *new_error_context_entry,
+ ParamListInfo *params);
#endif /* PARAMS_H */
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index 9ab4b5470b..ecc78d2ae6 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -202,6 +202,7 @@ extern int err_generic_string(int field, const char *str);
extern int geterrcode(void);
extern int geterrposition(void);
extern int getinternalerrposition(void);
+extern bool geterrshowstmt(void);
/*----------
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..47a5e31335
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,193 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /* emit error on parse stage */
+ paramValues[0] = (char *) "non-number -- parse";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT $1::int",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("text->int conversion error expected on parse stage, "
+ "got an error message from server: %s\n",
+ PQerrorMessage(conn));
+
+ /* emit error on bind stage */
+ paramValues[0] = (char *) "{malformed json -- bind :-{";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT (' ' || $1::text || ' ')::json",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Json parsing error expected on bind stage, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 0 params */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int /*exec*/",
+ 0, /* # params */
+ NULL, /* let the backend deduce param type */
+ NULL,
+ NULL,
+ NULL,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 3 params */
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ res = PQexecParams(
+ conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text /*exec*/",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1 /* ask for binary results */
+ );
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
On 2019-Sep-20, Andres Freund wrote:
+ appendStringInfoCharMacro(¶m_str, '\''); + for (p = pstring; *p; p++) + { + if (*p == '\'') /* double single quotes */ + appendStringInfoCharMacro(¶m_str, *p); + appendStringInfoCharMacro(¶m_str, *p); + } + appendStringInfoCharMacro(¶m_str, '\'');I know this is old code, but: This is really inefficient. Will cause a
lot of unnecessary memory-reallocations for large text outputs, because
we don't immediately grow to at least close to the required size.
I'd probably just count the ' in one pass, enlarge the stringinfo to the
required size, and then put the string directly into he stringbuffer.
Possibly just putting the necessary code into stringinfo.c. We already
have multiple copies of this inefficient logic...
I stole Alexey's code for this from downthread and tried to apply to all
these places. However, I did not put it to use in all these places you
mention, because each of them has slightly different requirements;
details below.
Now, I have to say that this doesn't make me terribly happy, because I
would like the additional ability to limit the printed values to N
bytes. This means the new function would have to have an additional
argument to indicate the maximum length (pass 0 to print all args
whole) ... and the logic then because more involved because we need
logic to stop printing early.
I think the current callers should all use the early termination logic;
having plpgsql potentially produce 1 MB of log output because of a long
argument seems silly. So I see no use for this function *without* the
length-limiting logic.
But even if not, I don't think writing data into the stringbuf directly
is that ugly, we do that in enough places that I'd argue that that's
basically part of how it's expected to be used.In HEAD there's at least
- postgres.c:errdetail_params(),
- pl_exec.c:format_preparedparamsdata()
pl_exec.c:format_expr_params()
These are changed in the patch; they're all alike.
- dblink.c:escape_param_str()
This one wants to use \' and \\ instead of just doubling each '.
The resulting string is used as libpq conninfo, so using doubled ' does
not work.
- deparse.c:deparseStringLiteral()
This one wants to use E''-style strings that ignore std_conforming_strings;
the logic would need to detect two chars ' and \ instead of just ' so
we'd need to use strcspn instead of strchr(); that seems a lot slower.
- xlog.c:do_pg_start_backup() (after "Add the escape character"),
This one wants to escape \n chars.
- tsearchcmds.c:serialize_deflist()
This wants E''-style strings, like deparse.c.
- ruleutils.c:simple_quote_literal()
Produce an escaped string according to the prevailing
std_conforming_strings.
I think it's possible to produce a function that would satisfy all of
these, but it would be another function similar to the one I'm writing
here, not the same one.
--
�lvaro Herrera https://www.2ndQuadrant.com/
ggPostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 99cf1d122bc935a4060c6b359fb2c262035039df Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--------
src/common/stringinfo.c | 40 ++++++++++++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 7 +++++++
src/pl/plpgsql/src/pl_exec.c | 34 ++++++++----------------------
4 files changed, 56 insertions(+), 35 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..0bff20ad67 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..f4dddf8223 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,46 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+ int len;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk, ensuring sufficient space for final bytes */
+ len = strlen(chunk_copy_start);
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+
+ /* add terminators */
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..7de2773e1f 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..23553a6948 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,10 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid));
paramno++;
}
@@ -8661,19 +8652,10 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]));
}
MemoryContextSwitchTo(oldcontext);
--
2.20.1
On 2019-Dec-03, Alvaro Herrera wrote:
Now, I have to say that this doesn't make me terribly happy, because I
would like the additional ability to limit the printed values to N
bytes. This means the new function would have to have an additional
argument to indicate the maximum length (pass 0 to print all args
whole) ... and the logic then because more involved because we need
logic to stop printing early.
So after actually writing this, it seems that it's better to have a
separate path for this; otherwise you need to scan the whole input
string (strchr or friends), which might be long. I tried to see how to
make strnlen() work for us, but I found no simple way to print the "..."
at the end. So this just runs once per char. I think that's okay,
since the intended usage is to print a few dozen bytes, like
ExecBuildSlotValueDescription does (64 bytes is what all its callers
do).
Another point: this implementation is not strict about how many
input characters it prints. It counts up to maxlen *output* characters,
which means if it duplicates a ', that's one less input char printed.
(Given the usage of this function, this seems a feature rather than a
bug. The first implementation did the opposite, and on the whole it
seemed worse.)
If anyone can suggest a smarter implementation, I'm all ears.
(Maybe do strnlen(maxlen), then count strnlen(1) starting at that point
-- so if that returns >=1, print the "..."?)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
(Maybe do strnlen(maxlen), then count strnlen(1) starting at that point
-- so if that returns >=1, print the "..."?)
So I found that I can make the code more reasonable with this simple
coding,
if (maxlen > 0)
{
s = pnstrdup(s, maxlen);
ellipsis = strnlen(s, maxlen + 1) > maxlen;
/* enlarge while we can do so cheaply */
enlargeStringInfo(str, maxlen);
}
... but the problem is that we now compile stringinfo.c for frontend
environments also, and there's no pnstrdup() in frontends. And to
introduce it, we'd need a configure check (because GNU libc has it) and
a src/port naive implementation and a fe_memutils.c addition.
Sigh.
Still, it's not that much code, so I'll just go do that and open a
separate thread for it.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-Dec-04, Alvaro Herrera wrote:
(Maybe do strnlen(maxlen), then count strnlen(1) starting at that point
-- so if that returns >=1, print the "..."?)So I found that I can make the code more reasonable with this simple
coding,
With strndup.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v3-0001-Add-strndup-pg_strndup.patchtext/x-diff; charset=us-asciiDownload
From 9a92076e53a6664ec61c5f475310c5d6edb91d90 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 4 Dec 2019 11:02:34 -0300
Subject: [PATCH v3 1/3] Add strndup / pg_strndup
The former is now a POSIX function. We already had pnstrdup, but it was
not offered to frontend code.
---
configure | 23 +++++++++++++++++++++++
configure.in | 3 ++-
src/common/fe_memutils.c | 26 ++++++++++++++++++++++++++
src/include/common/fe_memutils.h | 2 ++
src/include/pg_config.h.in | 7 +++++++
src/include/pg_config.h.win32 | 7 +++++++
src/include/port.h | 4 ++++
src/port/strndup.c | 32 ++++++++++++++++++++++++++++++++
8 files changed, 103 insertions(+), 1 deletion(-)
create mode 100644 src/port/strndup.c
diff --git a/configure b/configure
index 1d88983b34..fd8c759473 100755
--- a/configure
+++ b/configure
@@ -15482,6 +15482,16 @@ fi
cat >>confdefs.h <<_ACEOF
#define HAVE_DECL_STRLCPY $ac_have_decl
_ACEOF
+ac_fn_c_check_decl "$LINENO" "strndup" "ac_cv_have_decl_strndup" "$ac_includes_default"
+if test "x$ac_cv_have_decl_strndup" = xyes; then :
+ ac_have_decl=1
+else
+ ac_have_decl=0
+fi
+
+cat >>confdefs.h <<_ACEOF
+#define HAVE_DECL_STRNDUP $ac_have_decl
+_ACEOF
ac_fn_c_check_decl "$LINENO" "strnlen" "ac_cv_have_decl_strnlen" "$ac_includes_default"
if test "x$ac_cv_have_decl_strnlen" = xyes; then :
ac_have_decl=1
@@ -15815,6 +15825,19 @@ esac
fi
+ac_fn_c_check_func "$LINENO" "strndup" "ac_cv_func_strndup"
+if test "x$ac_cv_func_strndup" = xyes; then :
+ $as_echo "#define HAVE_STRNDUP 1" >>confdefs.h
+
+else
+ case " $LIBOBJS " in
+ *" strndup.$ac_objext "* ) ;;
+ *) LIBOBJS="$LIBOBJS strndup.$ac_objext"
+ ;;
+esac
+
+fi
+
ac_fn_c_check_func "$LINENO" "strnlen" "ac_cv_func_strnlen"
if test "x$ac_cv_func_strnlen" = xyes; then :
$as_echo "#define HAVE_STRNLEN 1" >>confdefs.h
diff --git a/configure.in b/configure.in
index a2cb20b5e3..11f66d19c3 100644
--- a/configure.in
+++ b/configure.in
@@ -1679,7 +1679,7 @@ AC_CHECK_DECLS(posix_fadvise, [], [], [#include <fcntl.h>])
]) # fi
AC_CHECK_DECLS(fdatasync, [], [], [#include <unistd.h>])
-AC_CHECK_DECLS([strlcat, strlcpy, strnlen])
+AC_CHECK_DECLS([strlcat, strlcpy, strndup, strnlen])
# This is probably only present on macOS, but may as well check always
AC_CHECK_DECLS(F_FULLFSYNC, [], [], [#include <fcntl.h>])
@@ -1738,6 +1738,7 @@ AC_REPLACE_FUNCS(m4_normalize([
srandom
strlcat
strlcpy
+ strndup
strnlen
strtof
]))
diff --git a/src/common/fe_memutils.c b/src/common/fe_memutils.c
index ce99b4f4da..896e5d46ca 100644
--- a/src/common/fe_memutils.c
+++ b/src/common/fe_memutils.c
@@ -101,6 +101,26 @@ pg_strdup(const char *in)
return tmp;
}
+char *
+pg_strndup(const char *in, size_t size)
+{
+ char *tmp;
+
+ if (!in)
+ {
+ fprintf(stderr,
+ _("cannot duplicate null pointer (internal error)\n"));
+ exit(EXIT_FAILURE);
+ }
+ tmp = strndup(in, size);
+ if (!tmp)
+ {
+ fprintf(stderr, _("out of memory\n"));
+ exit(EXIT_FAILURE);
+ }
+ return tmp;
+}
+
void
pg_free(void *ptr)
{
@@ -142,6 +162,12 @@ pstrdup(const char *in)
return pg_strdup(in);
}
+char *
+pnstrdup(const char *in, Size size)
+{
+ return pg_strndup(in, size);
+}
+
void *
repalloc(void *pointer, Size size)
{
diff --git a/src/include/common/fe_memutils.h b/src/include/common/fe_memutils.h
index a1e5940d31..c8797ffe38 100644
--- a/src/include/common/fe_memutils.h
+++ b/src/include/common/fe_memutils.h
@@ -23,6 +23,7 @@
* (except pg_malloc_extended with MCXT_ALLOC_NO_OOM)
*/
extern char *pg_strdup(const char *in);
+extern char *pg_strndup(const char *in, size_t size);
extern void *pg_malloc(size_t size);
extern void *pg_malloc0(size_t size);
extern void *pg_malloc_extended(size_t size, int flags);
@@ -31,6 +32,7 @@ extern void pg_free(void *pointer);
/* Equivalent functions, deliberately named the same as backend functions */
extern char *pstrdup(const char *in);
+extern char *pnstrdup(const char *in, Size size);
extern void *palloc(Size size);
extern void *palloc0(Size size);
extern void *palloc_extended(Size size, int flags);
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index c208dcdfc7..1db3f1b1e8 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -170,6 +170,10 @@
don't. */
#undef HAVE_DECL_STRLCPY
+/* Define to 1 if you have the declaration of `strndup', and to 0 if you
+ don't. */
+#undef HAVE_DECL_STRNDUP
+
/* Define to 1 if you have the declaration of `strnlen', and to 0 if you
don't. */
#undef HAVE_DECL_STRNLEN
@@ -545,6 +549,9 @@
/* Define to 1 if you have the `strlcpy' function. */
#undef HAVE_STRLCPY
+/* Define to 1 if you have the `strndup' function. */
+#undef HAVE_STRNDUP
+
/* Define to 1 if you have the `strnlen' function. */
#undef HAVE_STRNLEN
diff --git a/src/include/pg_config.h.win32 b/src/include/pg_config.h.win32
index 6c98ef4916..2ff2c3d4a8 100644
--- a/src/include/pg_config.h.win32
+++ b/src/include/pg_config.h.win32
@@ -129,6 +129,10 @@
don't. */
#define HAVE_DECL_RTLD_NOW 0
+/* Define to 1 if you have the declaration of `strndup', and to 0 if you
+ don't. */
+#undef HAVE_DECL_STRNDUP
+
/* Define to 1 if you have the declaration of `strnlen', and to 0 if you
don't. */
#define HAVE_DECL_STRNLEN 1
@@ -297,6 +301,9 @@
/* Define to 1 if you have the <pam/pam_appl.h> header file. */
/* #undef HAVE_PAM_PAM_APPL_H */
+/* Define to 1 if you have the `strndup' function. */
+#undef HAVE_STRNDUP
+
/* Define to 1 if you have the `strnlen' function. */
#define HAVE_STRNLEN 1
diff --git a/src/include/port.h b/src/include/port.h
index 10dcb5f0a6..e5d4486eb2 100644
--- a/src/include/port.h
+++ b/src/include/port.h
@@ -430,6 +430,10 @@ extern size_t strlcat(char *dst, const char *src, size_t siz);
extern size_t strlcpy(char *dst, const char *src, size_t siz);
#endif
+#if !HAVE_DECL_STRNDUP
+extern char *strndup(char *src, size_t siz);
+#endif
+
#if !HAVE_DECL_STRNLEN
extern size_t strnlen(const char *str, size_t maxlen);
#endif
diff --git a/src/port/strndup.c b/src/port/strndup.c
new file mode 100644
index 0000000000..eba41a0498
--- /dev/null
+++ b/src/port/strndup.c
@@ -0,0 +1,32 @@
+/*
+ * strndup.c
+ * Fallback implementation of strndup().
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/port/strndup.c
+ */
+
+#include "c.h"
+
+/*
+ * Implementation of posix' strndup for systems where it's not available.
+ */
+char *
+strndup(const char *str, size_t siz)
+{
+ char *dst;
+ int len;
+
+ len = strnlen(str, siz);
+ dst = malloc(len + 1);
+ if (dst == NULL)
+ return NULL;
+
+ memcpy(dst, str, len);
+ dst[len + 1] = '\0';
+
+ return dst;
+}
--
2.20.1
v3-0002-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From c8eea42dcd441fdf0c7ded15538bd113dfbf0ff7 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v3 2/3] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--------
src/common/stringinfo.c | 40 ++++++++++++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 7 +++++++
src/pl/plpgsql/src/pl_exec.c | 34 ++++++++----------------------
4 files changed, 56 insertions(+), 35 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..0bff20ad67 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..f4dddf8223 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,46 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+ int len;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk, ensuring sufficient space for final bytes */
+ len = strlen(chunk_copy_start);
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+
+ /* add terminators */
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..7de2773e1f 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..23553a6948 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,10 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid));
paramno++;
}
@@ -8661,19 +8652,10 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]));
}
MemoryContextSwitchTo(oldcontext);
--
2.20.1
v3-0003-Restrict-length-printed-by-appendStringInfoString.patchtext/x-diff; charset=us-asciiDownload
From a34b4728658891cb7d37433877efd0b706331dd3 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 18:12:25 -0300
Subject: [PATCH v3 3/3] Restrict length printed by
appendStringInfoStringQuoted
---
src/backend/tcop/postgres.c | 2 +-
src/common/stringinfo.c | 49 +++++++++++++++++++++------
src/include/lib/stringinfo.h | 3 +-
src/pl/plpgsql/src/pl_exec.c | 6 ++--
src/test/regress/expected/plpgsql.out | 14 ++++++++
src/test/regress/sql/plpgsql.sql | 13 +++++++
6 files changed, 72 insertions(+), 15 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0bff20ad67..c38ea2c743 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2363,7 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoStringQuoted(¶m_str, pstring);
+ appendStringInfoStringQuoted(¶m_str, pstring, 64);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index f4dddf8223..6f7555ef96 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -185,15 +185,34 @@ appendStringInfoString(StringInfo str, const char *s)
* and doubling all single quotes.
*/
void
-appendStringInfoStringQuoted(StringInfo str, const char *s)
+appendStringInfoStringQuoted(StringInfo str, const char *s, int maxlen)
{
- const char *chunk_search_start = s,
- *chunk_copy_start = s,
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
*chunk_end;
- int len;
+ bool ellipsis;
Assert(str != NULL);
+ if (maxlen > 0)
+ {
+ copy = pnstrdup(s, maxlen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = strnlen(s, maxlen + 1) > maxlen;
+ /* enlarge while we can do so cheaply */
+ enlargeStringInfo(str, maxlen);
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
appendStringInfoCharMacro(str, '\'');
while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
@@ -208,14 +227,22 @@ appendStringInfoStringQuoted(StringInfo str, const char *s)
chunk_search_start = chunk_end + 1;
}
- /* copy the last chunk, ensuring sufficient space for final bytes */
- len = strlen(chunk_copy_start);
- enlargeStringInfo(str, len + 2);
- appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
- /* add terminators */
- appendStringInfoCharMacro(str, '\'');
- str->data[str->len] = '\0';
+ /* ensure sufficient space for terminators */
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+ }
+
+ if (copy)
+ pfree(copy);
}
/*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index 7de2773e1f..956ccbba45 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -118,7 +118,8 @@ extern void appendStringInfoString(StringInfo str, const char *s);
* Append a null-terminated string to str, adding single quotes around it
* and doubling all single quotes.
*/
-extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s,
+ int maxlen);
/*------------------------
* appendStringInfoChar
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 23553a6948..752a66f379 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8614,7 +8614,8 @@ format_expr_params(PLpgSQL_execstate *estate,
appendStringInfoStringQuoted(¶mstr,
convert_value_to_string(estate,
paramdatum,
- paramtypeid));
+ paramtypeid),
+ 64);
paramno++;
}
@@ -8655,7 +8656,8 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
appendStringInfoStringQuoted(¶mstr,
convert_value_to_string(estate,
ppd->values[paramno],
- ppd->types[paramno]));
+ ppd->types[paramno]),
+ 64);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..a53ae28548 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que ...'
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
I made a number of changes on the proposed patch and ended up with the
attached. (I have not included the business to limit number of
characters, yet).
What this does is set an error callback, which adds the parameter values
in the DETAIL line. This is at odds with all existing error callbacks:
they only add stuff to the CONTEXT string. The implication is that
if an error site has a direct errdetail_log() and is run under this
error stack, that site's detail is going to be lost and replaced by the
"params" one. Currently there aren't many errdetail_log() lines, so it
doesn't seem like a terrible problem. However, it's something to keep
in mind.
This is not final form; there are a couple of XXX comments, and a number
of other small infelicities still to fix. But the overall idea should
be final.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v15-0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 1c5ffb2fb828a3589564d2f8af828e1be0549956 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v15 1/2] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--------
src/common/stringinfo.c | 40 ++++++++++++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 7 +++++++
src/pl/plpgsql/src/pl_exec.c | 34 ++++++++----------------------
4 files changed, 56 insertions(+), 35 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..0bff20ad67 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..f4dddf8223 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,46 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+ int len;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk, ensuring sufficient space for final bytes */
+ len = strlen(chunk_copy_start);
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+
+ /* add terminators */
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..7de2773e1f 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..23553a6948 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,10 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid));
paramno++;
}
@@ -8661,19 +8652,10 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]));
}
MemoryContextSwitchTo(oldcontext);
--
2.20.1
v15-0002-bashtanov-v14.patchtext/x-diff; charset=us-asciiDownload
From 850fb629ca2fa9de0868469ba9a122780462ce0f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 2 Dec 2019 16:02:48 -0300
Subject: [PATCH v15 2/2] bashtanov v14
---
doc/src/sgml/config.sgml | 23 +++
src/backend/nodes/params.c | 125 ++++++++++++
src/backend/tcop/postgres.c | 101 ++++-----
src/backend/utils/misc/guc.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/params.h | 3 +
src/include/utils/guc.h | 1 +
src/test/README | 3 +-
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpq5.c | 193 ++++++++++++++++++
11 files changed, 413 insertions(+), 50 deletions(-)
create mode 100644 src/test/examples/testlibpq5.c
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 53ac14490a..5d1c90282f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6595,6 +6595,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..dd0402a968 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,11 +15,14 @@
#include "postgres.h"
+#include "lib/stringinfo.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
+#include "utils/guc.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
/*
@@ -44,6 +47,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->paramValuesStr = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +62,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied, either.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -221,6 +227,8 @@ SerializeParamList(ParamListInfo paramLI, char **start_address)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied.
*/
ParamListInfo
RestoreParamList(char **start_address)
@@ -251,3 +259,120 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString - create a string to represent parameters list for
+ * logging and save it to params. If caller already knows textual
+ * representations for some or all parameters, it can pass an array of exactly
+ * params->numParams values as knownTextValues. It can contain NULLs for the
+ * individual values not known, or, if no text text values known at all the
+ * caller may pass NULL pointer.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ int paramStringLen = 0;
+ StringInfoData buf;
+ char **textValues;
+
+ /*
+ * nothing to do if it's is already set; also, no work if the param fetch
+ * hook is in use.
+ *
+ * XXX invoke the fetch hook to obtain the value? (probably useless)
+ */
+ if (params->paramValuesStr != NULL ||
+ params->paramFetch != NULL)
+ return;
+
+ initStringInfo(&buf);
+
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ textValues = (char **) palloc0(params->numParams * sizeof(char *));
+
+ /* calculate unknown text representations and pre-calculate byte lengths */
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData param = params->params[paramno];
+
+ /* add space for the param number of digits XXX slooow? */
+ for (int d = paramno + 1; d > 0; d /= 10)
+ paramStringLen++;
+
+ if (param.isnull)
+ {
+ paramStringLen += 4; /* for "NULL" */
+ }
+ else
+ {
+ /* find out textValues[paramno] */
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ {
+ textValues[paramno] = knownTextValues[paramno];
+ }
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+
+ getTypeOutputInfo(param.ptype, &typoutput, &typisvarlena);
+ textValues[paramno] = OidOutputFunctionCall(typoutput,
+ param.value);
+ }
+
+ /* Count the length of the text value itself */
+ paramStringLen += strlen(textValues[paramno]);
+ }
+ }
+
+ /*-----------------------------
+ * The output for each datum looks like this:
+ * "$1 = something, "
+ * ^ ^^^ ^^
+ * except for the last datum where we need \0 instead of ", ".
+ *-----------------------------
+ */
+ paramStringLen += params->numParams * 6 - 1;
+
+ MemoryContextSwitchTo(oldCxt);
+
+ /* enlarge at once to avoid multiple reallocations */
+ enlargeStringInfo(&buf, paramStringLen);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ appendStringInfo(&buf,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+ appendStringInfoStringQuoted(&buf, textValues[paramno]);
+ }
+
+ params->paramValuesStr = buf.data;
+
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * ParamsErrorCallback - callback for printing parameters in error context
+ */
+void
+ParamsErrorCallback(void *arg)
+{
+ ParamListInfo params = (ParamListInfo) arg;
+
+ if (params == NULL || params->paramValuesStr == NULL)
+ return;
+
+ /*
+ * XXX this clobbers any other DETAIL line that the error callsite could
+ * have had. Do we care?
+ */
+ errdetail_log("parameters: %s", params->paramValuesStr);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0bff20ad67..ef8eda6de8 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1613,6 +1613,7 @@ exec_bind_message(StringInfo input_message)
bool save_log_statement_stats = log_statement_stats;
bool snapshot_set = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1752,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ char **knownTextValues = NULL;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1819,9 +1822,26 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * Save the parameter string if we need it; otherwise free it.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ {
+ /*
+ * Allocate this on first use, making sure to put it
+ * in a short-lived context.
+ */
+ if (knownTextValues == NULL)
+ knownTextValues =
+ MemoryContextAllocZero(MessageContext,
+ numParams * sizeof(char *));
+ knownTextValues[paramno] = pstring;
+ }
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1871,6 +1891,9 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues);
}
else
params = NULL;
@@ -1878,6 +1901,12 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* Set the error callback so that parameters are logged, as needed */
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) params;
+ error_context_stack = ¶ms_errcxt;
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1923,6 +1952,12 @@ exec_bind_message(StringInfo input_message)
*/
PortalSetResultFormat(portal, numRFormats, rformats);
+ /*
+ * Done binding; remove the parameters error callback. Entries emitted
+ * later determine independently whether to log the parameters or not.
+ */
+ error_context_stack = error_context_stack->previous;
+
/*
* Send BindComplete.
*/
@@ -1979,6 +2014,7 @@ exec_execute_message(const char *portal_name, long max_rows)
bool execute_is_fetch;
bool was_logged = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2103,8 +2139,14 @@ exec_execute_message(const char *portal_name, long max_rows)
CHECK_FOR_INTERRUPTS();
/*
- * Okay to run the portal.
+ * Okay to run the portal. Set the error callback so that parameters are
+ * logged, as needed.
*/
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) portalParams;
+ error_context_stack = ¶ms_errcxt;
+
if (max_rows <= 0)
max_rows = FETCH_ALL;
@@ -2118,6 +2160,9 @@ exec_execute_message(const char *portal_name, long max_rows)
receiver->rDestroy(receiver);
+ /* Done executing; remove the params error callback */
+ error_context_stack = error_context_stack->previous;
+
if (completed)
{
if (is_xact_command)
@@ -2328,51 +2373,13 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (params && params->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
+ BuildParamLogString(params, NULL);
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoStringQuoted(¶m_str, pstring);
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params->paramValuesStr &&
+ params->paramValuesStr[0] != '\0')
+ errdetail("parameters: %s", params->paramValuesStr);
}
return 0;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ba74bf9f7d..8d951ce404 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9541879c1f..087190ce63 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..ba51dfe5e0 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *paramValuesStr; /* params as a single string for errors */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -156,5 +157,7 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues);
+extern void ParamsErrorCallback(void *arg);
#endif /* PARAMS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..47a5e31335
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,193 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /* emit error on parse stage */
+ paramValues[0] = (char *) "non-number -- parse";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT $1::int",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("text->int conversion error expected on parse stage, "
+ "got an error message from server: %s\n",
+ PQerrorMessage(conn));
+
+ /* emit error on bind stage */
+ paramValues[0] = (char *) "{malformed json -- bind :-{";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT (' ' || $1::text || ' ')::json",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Json parsing error expected on bind stage, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 0 params */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int /*exec*/",
+ 0, /* # params */
+ NULL, /* let the backend deduce param type */
+ NULL,
+ NULL,
+ NULL,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 3 params */
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ res = PQexecParams(
+ conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text /*exec*/",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1 /* ask for binary results */
+ );
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.20.1
Hi,
On 2019-12-05 14:19:45 -0300, Alvaro Herrera wrote:
What this does is set an error callback, which adds the parameter values
in the DETAIL line. This is at odds with all existing error callbacks:
they only add stuff to the CONTEXT string. The implication is that
if an error site has a direct errdetail_log() and is run under this
error stack, that site's detail is going to be lost and replaced by the
"params" one. Currently there aren't many errdetail_log() lines, so it
doesn't seem like a terrible problem. However, it's something to keep
in mind.
It'd not be too hard avoiding overwriting if we wanted. Makes me wonder
though, if we ought to invent something similar to the errdata fields we
have for schema, table, etc...
Haven't yet had time to catch up to the discussion in this thread of the
last two weeks...
Greetings,
Andres Freund
Hello
On 2019-Dec-05, Andres Freund wrote:
On 2019-12-05 14:19:45 -0300, Alvaro Herrera wrote:
What this does is set an error callback, which adds the parameter values
in the DETAIL line. This is at odds with all existing error callbacks:
they only add stuff to the CONTEXT string. The implication is that
if an error site has a direct errdetail_log() and is run under this
error stack, that site's detail is going to be lost and replaced by the
"params" one. Currently there aren't many errdetail_log() lines, so it
doesn't seem like a terrible problem. However, it's something to keep
in mind.It'd not be too hard avoiding overwriting if we wanted.
Hmm, maybe errdetail_log() ought to do that. However: if some place has
errdetail() and then this introduces errdetail_log(), then the former is
not included in the server log (only in the client log).
Makes me wonder though, if we ought to invent something similar to the
errdata fields we have for schema, table, etc...
Hmm, perhaps we should do that. It avoids the problem altogether.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-Dec-05, Alvaro Herrera wrote:
Makes me wonder though, if we ought to invent something similar to the
errdata fields we have for schema, table, etc...Hmm, perhaps we should do that. It avoids the problem altogether.
... then again, I'm not up for writing everything that would be required
to do that. If somebody wants to propose that separately, I volunteer
to review it. But let's not have that block this patch, since this is
already a clear improvement.
So here's v16. I got rid of all that strlen() business of the output
values; instead, preallocate a reasonable guess at the final length (64
bytes for each of the first 128 parameters, plus 16 bytes for for
parameter from 129 to 1024). This boils down to exactly the initial
size of a stringinfo (1024 bytes) when there are 8 parameters or less.
64 bytes avg of guessed length should be plenty (and it will be even
more so when I add the patch to limit the output length to 64+epsilon
bytes per param) I admit there are perhaps too many magical numbers in
those three lines, though.
I don't understand how we ended up with the params put in
errdetail_log() -- seems to have been introduced silently between v13
and v14. What's the reason for that? I put it back as errdetail().
Alexey added some discussion about using context/detail when he sent
that version, but I think that's an issue we can leave for the
hypothetical errparameters() patch.
One problem I noticed is that we don't log parameters when
exec_bind_message fetches the parameter values. So the very first
example problem in testlibpq5 fails to print the values of any
parameters previously seen. I don't think this is a real problem in
practice. You still get the unparseable value in the error message from
the input function, so not having the errdetail() does not seem very
important.
If anybody would like to review it once more, now's the time. I'm
aiming at getting it pushed tomorrow (including the length-limited
appendStringInfoStringQuoted stuff).
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v16-0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 1c5ffb2fb828a3589564d2f8af828e1be0549956 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v16 1/2] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--------
src/common/stringinfo.c | 40 ++++++++++++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 7 +++++++
src/pl/plpgsql/src/pl_exec.c | 34 ++++++++----------------------
4 files changed, 56 insertions(+), 35 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..0bff20ad67 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..f4dddf8223 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,46 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s)
+{
+ const char *chunk_search_start = s,
+ *chunk_copy_start = s,
+ *chunk_end;
+ int len;
+
+ Assert(str != NULL);
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk, ensuring sufficient space for final bytes */
+ len = strlen(chunk_copy_start);
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+
+ /* add terminators */
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..7de2773e1f 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,13 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..23553a6948 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,10 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid));
paramno++;
}
@@ -8661,19 +8652,10 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]));
}
MemoryContextSwitchTo(oldcontext);
--
2.20.1
v16-0002-Emit-parameter-values-during-query-bind-execute-.patchtext/x-diff; charset=us-asciiDownload
From fcbe0a5501271a5fa7b0e3f8611ae2ee896155af Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 2 Dec 2019 16:02:48 -0300
Subject: [PATCH v16 2/2] Emit parameter values during query bind/execute
errors
---
doc/src/sgml/config.sgml | 23 +++
src/backend/nodes/params.c | 121 +++++++++++
src/backend/tcop/postgres.c | 102 ++++-----
src/backend/utils/misc/guc.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/nodes/params.h | 3 +
src/include/utils/guc.h | 1 +
src/test/README | 3 +-
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpq5.c | 193 ++++++++++++++++++
11 files changed, 410 insertions(+), 50 deletions(-)
create mode 100644 src/test/examples/testlibpq5.c
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 53ac14490a..5d1c90282f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6595,6 +6595,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..13d90f6551 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,11 +15,14 @@
#include "postgres.h"
+#include "access/xact.h"
+#include "lib/stringinfo.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
/*
@@ -44,6 +47,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->paramValuesStr = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +62,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied, either.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -221,6 +227,8 @@ SerializeParamList(ParamListInfo paramLI, char **start_address)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied.
*/
ParamListInfo
RestoreParamList(char **start_address)
@@ -251,3 +259,116 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString
+ * Prepare for reporting parameter values.
+ *
+ * Create a string to represent parameters list for logging and save it to
+ * params. If caller already knows textual representations for some or all
+ * parameters, it can pass an array of exactly params->numParams values as
+ * knownTextValues. It can contain NULLs for the individual values not known,
+ * or, if no text text values known at all the caller may pass NULL pointer.
+ *
+ * This should only be called if log_parameters_on_error is set, or caller
+ * wants to report the query parameters for other reasons.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ StringInfoData buf;
+
+ /*
+ * Nothing to do if it's is already set; also, no work if the param fetch
+ * hook is in use. (Mostly because it's unimplemented ... but that's
+ * because it doesn't seem a case worth pursuing.) Also, avoid calling
+ * user-defined I/O functions when in an aborted transaction. It might
+ * be possible to improve on this when some knownTextValues exist.
+ */
+ if (params->paramValuesStr != NULL ||
+ params->paramFetch != NULL ||
+ IsAbortedTransactionBlockState())
+ return;
+
+ /*
+ * Initialize the output stringinfo, in the callers memory context.
+ *
+ * To avoid repeated stringinfo enlargement, we assume 64 bytes per param
+ * for the first 128 params, then 16 bytes for each until completing 1024.
+ * Add zero bytes for parameters beyond 1024, since things are difficult
+ * to anticipate at that point. Note that because stringinfo initial size
+ * is 1024, we don't have to do anything with 8 parameters or less.
+ *
+ * (It doesn't seem worth the time or code complication to strlen() each
+ * parameter individually.)
+ */
+ initStringInfo(&buf);
+ if (params->numParams > 8)
+ enlargeStringInfo(&buf,
+ 64 * Min(params->numParams, 128) +
+ 16 * Min(Max(0, params->numParams - 128), 1024 - 128));
+
+ /* Use a temporary context to call output functions, just in case */
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_SMALL_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *param = ¶ms->params[paramno];
+
+ appendStringInfo(&buf,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (param->isnull || !OidIsValid(param->ptype))
+ appendStringInfoString(&buf, "NULL");
+ else
+ {
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ appendStringInfoStringQuoted(&buf, knownTextValues[paramno]);
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ char *pstring;
+
+ getTypeOutputInfo(param->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, param->value);
+ appendStringInfoStringQuoted(&buf, pstring);
+ }
+ }
+
+ MemoryContextReset(tmpCxt);
+ }
+
+ params->paramValuesStr = buf.data;
+
+ MemoryContextSwitchTo(oldCxt);
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * ParamsErrorCallback - callback for printing parameters in error context
+ *
+ * Note that this is a no-op unless BuildParamLogString has been called
+ * beforehand.
+ */
+void
+ParamsErrorCallback(void *arg)
+{
+ ParamListInfo params = (ParamListInfo) arg;
+
+ if (params == NULL || params->paramValuesStr == NULL)
+ return;
+
+ /*
+ * XXX this clobbers any other DETAIL line that the error callsite could
+ * have had. Do we care?
+ */
+ errdetail("parameters: %s", params->paramValuesStr);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0bff20ad67..9a009a5a9b 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1613,6 +1613,7 @@ exec_bind_message(StringInfo input_message)
bool save_log_statement_stats = log_statement_stats;
bool snapshot_set = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1752,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ char **knownTextValues = NULL;
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1819,9 +1822,26 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * Save the parameter string if we need it; otherwise free it.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ {
+ /*
+ * Allocate this on first use, making sure to put it
+ * in a short-lived context.
+ */
+ if (knownTextValues == NULL)
+ knownTextValues =
+ MemoryContextAllocZero(MessageContext,
+ numParams * sizeof(char *));
+ knownTextValues[paramno] = pstring;
+ }
+ else if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1871,6 +1891,10 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /* Prepare for printing parameters, if configured to do so */
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues);
}
else
params = NULL;
@@ -1878,6 +1902,12 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* Set the error callback so that parameters are logged, as needed */
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) params;
+ error_context_stack = ¶ms_errcxt;
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1923,6 +1953,12 @@ exec_bind_message(StringInfo input_message)
*/
PortalSetResultFormat(portal, numRFormats, rformats);
+ /*
+ * Done binding; remove the parameters error callback. Entries emitted
+ * later determine independently whether to log the parameters or not.
+ */
+ error_context_stack = error_context_stack->previous;
+
/*
* Send BindComplete.
*/
@@ -1979,6 +2015,7 @@ exec_execute_message(const char *portal_name, long max_rows)
bool execute_is_fetch;
bool was_logged = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2103,8 +2140,14 @@ exec_execute_message(const char *portal_name, long max_rows)
CHECK_FOR_INTERRUPTS();
/*
- * Okay to run the portal.
+ * Okay to run the portal. Set the error callback so that parameters are
+ * logged. The parameters must have been saved during the bind phase.
*/
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) portalParams;
+ error_context_stack = ¶ms_errcxt;
+
if (max_rows <= 0)
max_rows = FETCH_ALL;
@@ -2118,6 +2161,9 @@ exec_execute_message(const char *portal_name, long max_rows)
receiver->rDestroy(receiver);
+ /* Done executing; remove the params error callback */
+ error_context_stack = error_context_stack->previous;
+
if (completed)
{
if (is_xact_command)
@@ -2328,51 +2374,13 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (params && params->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
+ BuildParamLogString(params, NULL);
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoStringQuoted(¶m_str, pstring);
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params->paramValuesStr &&
+ params->paramValuesStr[0] != '\0')
+ errdetail("parameters: %s", params->paramValuesStr);
}
return 0;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ba74bf9f7d..8d951ce404 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9541879c1f..087190ce63 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..ba51dfe5e0 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *paramValuesStr; /* params as a single string for errors */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -156,5 +157,7 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues);
+extern void ParamsErrorCallback(void *arg);
#endif /* PARAMS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..47a5e31335
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,193 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /* emit error on parse stage */
+ paramValues[0] = (char *) "non-number -- parse";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT $1::int",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("text->int conversion error expected on parse stage, "
+ "got an error message from server: %s\n",
+ PQerrorMessage(conn));
+
+ /* emit error on bind stage */
+ paramValues[0] = (char *) "{malformed json -- bind :-{";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT (' ' || $1::text || ' ')::json",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Json parsing error expected on bind stage, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 0 params */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int /*exec*/",
+ 0, /* # params */
+ NULL, /* let the backend deduce param type */
+ NULL,
+ NULL,
+ NULL,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 3 params */
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ res = PQexecParams(
+ conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text /*exec*/",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1 /* ask for binary results */
+ );
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.20.1
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
If anybody would like to review it once more, now's the time. I'm
aiming at getting it pushed tomorrow (including the length-limited
appendStringInfoStringQuoted stuff).
Um, surely this bit is unacceptable?
+ /*
+ * XXX this clobbers any other DETAIL line that the error callsite could
+ * have had. Do we care?
+ */
+ errdetail("parameters: %s", params->paramValuesStr);
Even if the scope of the clobber were limited to the parameter input
conversion step, this seems like a pretty bad compromise. But AFAICS
this'll clobber errdetail for *any* error occurring during query
execution in extended query mode. Not only does that render useless
a whole lot of sweat we've put into detail messages over the years,
but it means that you get fundamentally different (and largely worse)
error reporting in extended query mode than you do in simple query mode.
I think this issue has to be fixed before committing, not after.
The most straightforward fix would be to make the parameter values
a brand new component of error reports. However, I see the point
that that'd require a huge amount of additional work, and cooperation
of client-side libraries that might be a long time coming.
Possibly a workable compromise is to emit the info as an error
context line, appending it to whatever context exists today.
The result might look like, say,
ERROR: whatever
CONTEXT: SQL function "foo"
extended query with parameters x, y, ...
For extra credit maybe we could include the query's statement or
portal name?
errcontext("extended query \"%s\" with parameters %s", ...);
An independent point: it seems like just wishful thinking to imagine that
src/test/examples/ can serve as a regression test for this. Nor is the
proposed program very useful as an example. I'd drop that and find a
way to have an actual, routinely-exercised-by-check-world regression
test. If memory serves, pgbench can be cajoled into executing stuff
in extended query mode --- maybe we could create a test case using
that?
regards, tom lane
On 2019-Dec-05, Tom Lane wrote:
Possibly a workable compromise is to emit the info as an error
context line, appending it to whatever context exists today.
The result might look like, say,ERROR: whatever
CONTEXT: SQL function "foo"
extended query with parameters x, y, ...
This is easy to accomodate -- just change the errdetail() to errcontext.
That makes the params be reported different than in the non-error case
(for duration).
For extra credit maybe we could include the query's statement or
portal name?errcontext("extended query \"%s\" with parameters %s", ...);
Sure. Done in the attached.
An independent point: it seems like just wishful thinking to imagine that
src/test/examples/ can serve as a regression test for this. Nor is the
proposed program very useful as an example. I'd drop that and find a
way to have an actual, routinely-exercised-by-check-world regression
test. If memory serves, pgbench can be cajoled into executing stuff
in extended query mode --- maybe we could create a test case using
that?
I tried
pgbench -c1 -t1 -n -M prepared -f errparameters.bnch
with this file
select '{ invalid ' as value \gset
select column1::jsonb from (values (:value)) as q;
and got these lines:
2019-12-06 13:49:36.388 -03 [31028] ERROR: invalid input syntax for type json
2019-12-06 13:49:36.388 -03 [31028] DETAIL: Token "invalid" is invalid.
2019-12-06 13:49:36.388 -03 [31028] CONTEXT: JSON data, line 1: { invalid...
extended query with parameters: $1 = '{ invalid '
2019-12-06 13:49:36.388 -03 [31028] STATEMENT: select column1::jsonb from (values ($1)) as q;
With this file,
select '1' as one \gset
SELECT 1 / (random() / 2)::int, :one::int, :two::int;
using the same pgbench invocation as above, I get this in the log:
2019-12-06 14:50:59.181 -03 [6187] ERROR: division by zero
2019-12-06 14:50:59.181 -03 [6187] CONTEXT: extended query with parameters: $1 = '1', $2 = NULL
2019-12-06 14:50:59.181 -03 [6187] STATEMENT: SELECT 1 / (random() / 2)::int, $1::int, $2::int;
(While testing this I noticed that failing to strdup the text repr of
the datum when it's given as a text-format parameter results in bogus
output. So I added a pstrdup there.)
(It seems a bit weird that if I assign NULL to :two pgbench stores the
empty string, instead of the NULL that I get as in the above which is
what happens when the variable is not defined at all. That's probably a
bug in \gset ...)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
errcontext.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index 13d90f6551..a881df578e 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -23,6 +23,7 @@
#include "utils/datum.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/portal.h"
/*
@@ -361,14 +362,17 @@ BuildParamLogString(ParamListInfo params, char **knownTextValues)
void
ParamsErrorCallback(void *arg)
{
- ParamListInfo params = (ParamListInfo) arg;
+ ParamsErrorCbData *data = (ParamsErrorCbData *) arg;
- if (params == NULL || params->paramValuesStr == NULL)
+ if (data == NULL ||
+ data->params == NULL ||
+ data->params->paramValuesStr == NULL)
return;
- /*
- * XXX this clobbers any other DETAIL line that the error callsite could
- * have had. Do we care?
- */
- errdetail("parameters: %s", params->paramValuesStr);
+ if (data->portalName && data->portalName[0] != '\0')
+ errcontext("extended query \"%s\" with parameters: %s",
+ data->portalName, data->params->paramValuesStr);
+ else
+ errcontext("extended query with parameters: %s",
+ data->params->paramValuesStr);
}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 9a009a5a9b..05a03ae924 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1753,7 +1753,7 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
- char **knownTextValues = NULL;
+ char **knownTextValues = NULL; /* allocate on first use */
params = makeParamList(numParams);
@@ -1822,22 +1822,19 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /*
- * Save the parameter string if we need it; otherwise free it.
- */
+ /* When logging params, save a copy for later. */
if (pstring)
{
if (log_parameters_on_error)
{
- /*
- * Allocate this on first use, making sure to put it
- * in a short-lived context.
- */
if (knownTextValues == NULL)
knownTextValues =
MemoryContextAllocZero(MessageContext,
numParams * sizeof(char *));
- knownTextValues[paramno] = pstring;
+ knownTextValues[paramno] =
+ pstring == pbuf.data ?
+ MemoryContextStrdup(MessageContext, pstring) :
+ pstring;
}
else if (pstring != pbuf.data)
pfree(pstring);
@@ -1905,7 +1902,8 @@ exec_bind_message(StringInfo input_message)
/* Set the error callback so that parameters are logged, as needed */
params_errcxt.previous = error_context_stack;
params_errcxt.callback = ParamsErrorCallback;
- params_errcxt.arg = (void *) params;
+ params_errcxt.arg = (void *) &((ParamsErrorCbData)
+ { portal->name, params });
error_context_stack = ¶ms_errcxt;
/* Get the result format codes */
@@ -2145,7 +2143,8 @@ exec_execute_message(const char *portal_name, long max_rows)
*/
params_errcxt.previous = error_context_stack;
params_errcxt.callback = ParamsErrorCallback;
- params_errcxt.arg = (void *) portalParams;
+ params_errcxt.arg = (void *) &((ParamsErrorCbData)
+ { portal->name, portalParams });
error_context_stack = ¶ms_errcxt;
if (max_rows <= 0)
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index ba51dfe5e0..1e557ca7fd 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -150,6 +150,12 @@ typedef struct ParamExecData
bool isnull;
} ParamExecData;
+/* type of argument for ParamsErrorCallback */
+typedef struct ParamsErrorCbData
+{
+ const char *portalName;
+ ParamListInfo params;
+} ParamsErrorCbData;
/* Functions found in src/backend/nodes/params.c */
extern ParamListInfo makeParamList(int numParams);
pá 6. 12. 2019 v 18:57 odesílatel Alvaro Herrera <alvherre@2ndquadrant.com>
napsal:
On 2019-Dec-05, Tom Lane wrote:
Possibly a workable compromise is to emit the info as an error
context line, appending it to whatever context exists today.
The result might look like, say,ERROR: whatever
CONTEXT: SQL function "foo"
extended query with parameters x, y, ...This is easy to accomodate -- just change the errdetail() to errcontext.
That makes the params be reported different than in the non-error case
(for duration).For extra credit maybe we could include the query's statement or
portal name?errcontext("extended query \"%s\" with parameters %s", ...);
Sure. Done in the attached.
An independent point: it seems like just wishful thinking to imagine that
src/test/examples/ can serve as a regression test for this. Nor is the
proposed program very useful as an example. I'd drop that and find a
way to have an actual, routinely-exercised-by-check-world regression
test. If memory serves, pgbench can be cajoled into executing stuff
in extended query mode --- maybe we could create a test case using
that?I tried
pgbench -c1 -t1 -n -M prepared -f errparameters.bnch
with this fileselect '{ invalid ' as value \gset
select column1::jsonb from (values (:value)) as q;and got these lines:
2019-12-06 13:49:36.388 -03 [31028] ERROR: invalid input syntax for type
json
2019-12-06 13:49:36.388 -03 [31028] DETAIL: Token "invalid" is invalid.
2019-12-06 13:49:36.388 -03 [31028] CONTEXT: JSON data, line 1: {
invalid...
extended query with parameters: $1 = '{ invalid '
2019-12-06 13:49:36.388 -03 [31028] STATEMENT: select column1::jsonb from
(values ($1)) as q;With this file,
select '1' as one \gset
SELECT 1 / (random() / 2)::int, :one::int, :two::int;using the same pgbench invocation as above, I get this in the log:
2019-12-06 14:50:59.181 -03 [6187] ERROR: division by zero
2019-12-06 14:50:59.181 -03 [6187] CONTEXT: extended query with
parameters: $1 = '1', $2 = NULL
2019-12-06 14:50:59.181 -03 [6187] STATEMENT: SELECT 1 / (random() /
2)::int, $1::int, $2::int;(While testing this I noticed that failing to strdup the text repr of
the datum when it's given as a text-format parameter results in bogus
output. So I added a pstrdup there.)(It seems a bit weird that if I assign NULL to :two pgbench stores the
empty string, instead of the NULL that I get as in the above which is
what happens when the variable is not defined at all. That's probably a
bug in \gset ...)
psql session variables cannot to hold NULL
Regards
Pavel
Show quoted text
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
I added some tests to the pgbench suite in the attached. I also finally
put it the business to limit the length of parameters reported.
I'd probably drop testlibpq5.c, since it seems a bit pointless now ...
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v17-0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 6e0a777cb51bcb17e15977fbc729fef6b80b3aa8 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v17 1/2] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +---
src/common/stringinfo.c | 67 +++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 8 ++++
src/pl/plpgsql/src/pl_exec.c | 36 ++++----------
src/test/regress/expected/plpgsql.out | 14 ++++++
src/test/regress/sql/plpgsql.sql | 13 ++++++
6 files changed, 113 insertions(+), 35 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..3d3172e83d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring, 0);
pfree(pstring);
}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..6f7555ef96 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -178,6 +178,73 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s, int maxlen)
+{
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
+ *chunk_end;
+ bool ellipsis;
+
+ Assert(str != NULL);
+
+ if (maxlen > 0)
+ {
+ copy = pnstrdup(s, maxlen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = strnlen(s, maxlen + 1) > maxlen;
+ /* enlarge while we can do so cheaply */
+ enlargeStringInfo(str, maxlen);
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
+
+ /* ensure sufficient space for terminators */
+ enlargeStringInfo(str, len + 2);
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ str->data[str->len] = '\0';
+ }
+
+ if (copy)
+ pfree(copy);
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..956ccbba45 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,14 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, const char *s,
+ int maxlen);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..c3ae409f39 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,11 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid),
+ 0);
paramno++;
}
@@ -8661,19 +8653,11 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]),
+ 0);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..cd2c79f4d5 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que mirase bien lo que hacia?'''
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
v17-0002-Emit-parameter-values-during-query-bind-execute-.patchtext/x-diff; charset=us-asciiDownload
From a898bbfb4aefa98325cdd2af9a1b0627c5e7f745 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 2 Dec 2019 16:02:48 -0300
Subject: [PATCH v17 2/2] Emit parameter values during query bind/execute
errors
---
doc/src/sgml/config.sgml | 23 +++
src/backend/nodes/params.c | 130 ++++++++++++
src/backend/tcop/postgres.c | 110 +++++-----
src/backend/utils/misc/guc.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pgbench/t/001_pgbench_with_server.pl | 35 ++++
src/include/nodes/params.h | 10 +
src/include/utils/guc.h | 1 +
src/test/README | 3 +-
src/test/examples/.gitignore | 1 +
src/test/examples/Makefile | 2 +-
src/test/examples/testlibpq5.c | 193 ++++++++++++++++++
12 files changed, 469 insertions(+), 50 deletions(-)
create mode 100644 src/test/examples/testlibpq5.c
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 53ac14490a..5d1c90282f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6595,6 +6595,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..cbdaec9b7c 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,11 +15,15 @@
#include "postgres.h"
+#include "access/xact.h"
+#include "lib/stringinfo.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+#include "utils/portal.h"
/*
@@ -44,6 +48,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->paramValuesStr = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +63,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied, either.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -158,6 +165,8 @@ EstimateParamListSpace(ParamListInfo paramLI)
* RestoreParamList can be used to recreate a ParamListInfo based on the
* serialized representation; this will be a static, self-contained copy
* just as copyParamList would create.
+ *
+ * paramValuesStr is not included.
*/
void
SerializeParamList(ParamListInfo paramLI, char **start_address)
@@ -251,3 +260,124 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString
+ * Prepare for reporting parameter values.
+ *
+ * Create a string to represent parameters list for logging and save it to
+ * params. If caller already knows textual representations for some or all
+ * parameters, it can pass an array of exactly params->numParams values as
+ * knownTextValues. It can contain NULLs for the individual values not known,
+ * or, if no text text values known at all the caller may pass NULL pointer.
+ *
+ * If valueLen is not zero, that's the maximum number of characters of the
+ * input string printed; an ellipsis is added if more characters exist.
+ * (Added quotes are not considered.)
+ *
+ * This should only be called if log_parameters_on_error is set, or caller
+ * wants to report the query parameters for other reasons.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues, int valueLen)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ StringInfoData buf;
+
+ /*
+ * Nothing to do if it's is already set; also, no work if the param fetch
+ * hook is in use. (Mostly because it's unimplemented ... but that's
+ * because it doesn't seem a case worth pursuing.) Also, avoid calling
+ * user-defined I/O functions when in an aborted transaction. It might
+ * be possible to improve on this when some knownTextValues exist.
+ */
+ if (params->paramValuesStr != NULL ||
+ params->paramFetch != NULL ||
+ IsAbortedTransactionBlockState())
+ return;
+
+ /*
+ * Initialize the output stringinfo, in the callers memory context.
+ *
+ * To avoid repeated stringinfo enlargement, we assume 64 bytes per param
+ * for the first 128 params, then 16 bytes for each until completing 1024.
+ * Add zero bytes for parameters beyond 1024, since things are difficult
+ * to anticipate at that point. Note that because stringinfo initial size
+ * is 1024, we don't have to do anything with 8 parameters or less.
+ *
+ * (It doesn't seem worth the time or code complication to strlen() each
+ * parameter individually.)
+ */
+ initStringInfo(&buf);
+ if (params->numParams > 8)
+ enlargeStringInfo(&buf,
+ 64 * Min(params->numParams, 128) +
+ 16 * Min(Max(0, params->numParams - 128), 1024 - 128));
+
+ /* Use a temporary context to call output functions, just in case */
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_SMALL_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *param = ¶ms->params[paramno];
+
+ appendStringInfo(&buf,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (param->isnull || !OidIsValid(param->ptype))
+ appendStringInfoString(&buf, "NULL");
+ else
+ {
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ appendStringInfoStringQuoted(&buf, knownTextValues[paramno],
+ valueLen);
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ char *pstring;
+
+ getTypeOutputInfo(param->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, param->value);
+ appendStringInfoStringQuoted(&buf, pstring, valueLen);
+ }
+ }
+
+ MemoryContextReset(tmpCxt);
+ }
+
+ params->paramValuesStr = buf.data;
+
+ MemoryContextSwitchTo(oldCxt);
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * ParamsErrorCallback - callback for printing parameters in error context
+ *
+ * Note that this is a no-op unless BuildParamLogString has been called
+ * beforehand.
+ */
+void
+ParamsErrorCallback(void *arg)
+{
+ ParamsErrorCbData *data = (ParamsErrorCbData *) arg;
+
+ if (data == NULL ||
+ data->params == NULL ||
+ data->params->paramValuesStr == NULL)
+ return;
+
+ if (data->portalName && data->portalName[0] != '\0')
+ errcontext("extended query \"%s\" with parameters: %s",
+ data->portalName, data->params->paramValuesStr);
+ else
+ errcontext("extended query with parameters: %s",
+ data->params->paramValuesStr);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3d3172e83d..1d23ec7746 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1613,6 +1613,7 @@ exec_bind_message(StringInfo input_message)
bool save_log_statement_stats = log_statement_stats;
bool snapshot_set = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1752,6 +1753,8 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ char **knownTextValues = NULL; /* allocate on first use */
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1819,9 +1822,28 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /* When logging params, save a copy for later */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ {
+ MemoryContext oldcxt;
+
+ oldcxt = MemoryContextSwitchTo(MessageContext);
+ if (knownTextValues == NULL)
+ knownTextValues =
+ palloc0(numParams * sizeof(char *));
+ /*
+ * Note: must copy at least one more byte than
+ * BuildParamLogString wants to see; otherwise it'll
+ * fail to include the ellipsis ...
+ */
+ knownTextValues[paramno] = pnstrdup(pstring, 65);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1871,6 +1893,14 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * Prepare for printing parameters, if configured to do so. (This is
+ * saved in the portal, so that they'll appear when the query is
+ * executed later.)
+ */
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues, 64);
}
else
params = NULL;
@@ -1878,6 +1908,13 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* Set the error callback so that parameters are logged, as needed */
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) &((ParamsErrorCbData)
+ { portal->name, params });
+ error_context_stack = ¶ms_errcxt;
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1923,6 +1960,12 @@ exec_bind_message(StringInfo input_message)
*/
PortalSetResultFormat(portal, numRFormats, rformats);
+ /*
+ * Done binding; remove the parameters error callback. Entries emitted
+ * later determine independently whether to log the parameters or not.
+ */
+ error_context_stack = error_context_stack->previous;
+
/*
* Send BindComplete.
*/
@@ -1979,6 +2022,7 @@ exec_execute_message(const char *portal_name, long max_rows)
bool execute_is_fetch;
bool was_logged = false;
char msec_str[32];
+ ErrorContextCallback params_errcxt;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2103,8 +2147,15 @@ exec_execute_message(const char *portal_name, long max_rows)
CHECK_FOR_INTERRUPTS();
/*
- * Okay to run the portal.
+ * Okay to run the portal. Set the error callback so that parameters are
+ * logged. The parameters must have been saved during the bind phase.
*/
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) &((ParamsErrorCbData)
+ { portal->name, portalParams });
+ error_context_stack = ¶ms_errcxt;
+
if (max_rows <= 0)
max_rows = FETCH_ALL;
@@ -2118,6 +2169,9 @@ exec_execute_message(const char *portal_name, long max_rows)
receiver->rDestroy(receiver);
+ /* Done executing; remove the params error callback */
+ error_context_stack = error_context_stack->previous;
+
if (completed)
{
if (is_xact_command)
@@ -2328,51 +2382,13 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (params && params->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
+ BuildParamLogString(params, NULL, 0);
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoStringQuoted(¶m_str, pstring, 0);
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params->paramValuesStr &&
+ params->paramValuesStr[0] != '\0')
+ errdetail("parameters: %s", params->paramValuesStr);
}
return 0;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ba74bf9f7d..8d951ce404 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9541879c1f..087190ce63 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 1845869016..021d7a1826 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -270,6 +270,41 @@ COMMIT;
}
});
+# Verify server logging of parameters
+$node->append_conf('postgresql.conf', "log_parameters_on_error = true");
+$node->reload;
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: division by zero},
+ qr{CONTEXT: extended query with parameters: \$1 = '1', \$2 = NULL}
+ ],
+ 'server parameter logging',
+ {
+ '001_param_1' => q{select '1' as one \gset
+SELECT 1 / (random() / 2)::int, :one::int, :two::int;
+}
+ });
+
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: invalid input syntax for type json},
+ qr[CONTEXT: JSON data, line 1: \{ invalid\.\.\.
+extended query with parameters: \$1 = '\{ invalid ', \$2 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que ...'$]m
+ ],
+ 'server parameter logging',
+ {
+ '001_param_2' => q[select '{ invalid ' as value \gset
+select $$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$$ as long \gset
+select column1::jsonb from (values (:value), (:long)) as q;
+]
+ });
+
# test expressions
# command 1..3 and 23 depend on random seed which is used to call srandom.
pgbench(
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..d7a7e1f191 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *paramValuesStr; /* params as a single string for errors */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -149,6 +150,12 @@ typedef struct ParamExecData
bool isnull;
} ParamExecData;
+/* type of argument for ParamsErrorCallback */
+typedef struct ParamsErrorCbData
+{
+ const char *portalName;
+ ParamListInfo params;
+} ParamsErrorCbData;
/* Functions found in src/backend/nodes/params.c */
extern ParamListInfo makeParamList(int numParams);
@@ -156,5 +163,8 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues,
+ int valueLen);
+extern void ParamsErrorCallback(void *arg);
#endif /* PARAMS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
diff --git a/src/test/README b/src/test/README
index b5ccfc0cf6..22e624d8ba 100644
--- a/src/test/README
+++ b/src/test/README
@@ -12,8 +12,7 @@ authentication/
Tests for authentication
examples/
- Demonstration programs for libpq that double as regression tests via
- "make check"
+ Demonstration programs for libpq that double as regression tests
isolation/
Tests for concurrent behavior at the SQL level
diff --git a/src/test/examples/.gitignore b/src/test/examples/.gitignore
index 1957ec198f..007aaee590 100644
--- a/src/test/examples/.gitignore
+++ b/src/test/examples/.gitignore
@@ -2,5 +2,6 @@
/testlibpq2
/testlibpq3
/testlibpq4
+/testlibpq5
/testlo
/testlo64
diff --git a/src/test/examples/Makefile b/src/test/examples/Makefile
index a67f456904..0407ca60bc 100644
--- a/src/test/examples/Makefile
+++ b/src/test/examples/Makefile
@@ -14,7 +14,7 @@ override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
LDFLAGS_INTERNAL += $(libpq_pgport)
-PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlo testlo64
+PROGS = testlibpq testlibpq2 testlibpq3 testlibpq4 testlibpq5 testlo testlo64
all: $(PROGS)
diff --git a/src/test/examples/testlibpq5.c b/src/test/examples/testlibpq5.c
new file mode 100644
index 0000000000..47a5e31335
--- /dev/null
+++ b/src/test/examples/testlibpq5.c
@@ -0,0 +1,193 @@
+/*
+ * src/test/examples/testlibpq5.c
+ *
+ * testlibpq5.c
+ * Test logging of statement parameters in case of errors.
+ */
+
+#ifdef WIN32
+#include <windows.h>
+#endif
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <sys/types.h>
+#include "libpq-fe.h"
+
+static void
+exit_nicely(PGconn *conn)
+{
+ PQfinish(conn);
+ exit(1);
+}
+
+int
+main(int argc, char **argv)
+{
+ const char *conninfo;
+ PGconn *conn;
+ PGresult *res;
+ const char *paramValues[3];
+ int paramLengths[3];
+ int paramFormats[3];
+ uint32_t binaryIntVal;
+
+ /*
+ * If the user supplies a parameter on the command line, use it as the
+ * conninfo string; otherwise default to setting dbname=postgres and using
+ * environment variables or defaults for all other connection parameters.
+ */
+ if (argc > 1)
+ conninfo = argv[1];
+ else
+ conninfo = "dbname = postgres";
+
+ /* Make a connection to the database */
+ conn = PQconnectdb(conninfo);
+
+ /* Check to see that the backend connection was successfully made */
+ if (PQstatus(conn) != CONNECTION_OK)
+ {
+ fprintf(stderr, "Connection to database failed: %s",
+ PQerrorMessage(conn));
+ exit_nicely(conn);
+ }
+
+ /* Set always-secure search path, so malicious users can't take control. */
+ res = PQexec(conn,
+ "SELECT pg_catalog.set_config('search_path', '', false)");
+ if (PQresultStatus(res) != PGRES_TUPLES_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /*
+ * Transmit parameters in different forms and make a statement fail. User
+ * can then verify the server log.
+ */
+ res = PQexec(conn, "SET log_parameters_on_error = on");
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ {
+ fprintf(stderr, "SET failed: %s", PQerrorMessage(conn));
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+
+ /* emit error on parse stage */
+ paramValues[0] = (char *) "non-number -- parse";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT $1::int",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("text->int conversion error expected on parse stage, "
+ "got an error message from server: %s\n",
+ PQerrorMessage(conn));
+
+ /* emit error on bind stage */
+ paramValues[0] = (char *) "{malformed json -- bind :-{";
+ paramLengths[0] = strlen(paramValues[2]) + 1;
+ paramFormats[0] = 0; /* text */
+ res = PQexecParams(conn,
+ "SELECT (' ' || $1::text || ' ')::json",
+ 1, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Json parsing error expected on bind stage, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 0 params */
+ res = PQexecParams(conn,
+ "SELECT 1 / (random() / 2)::int /*exec*/",
+ 0, /* # params */
+ NULL, /* let the backend deduce param type */
+ NULL,
+ NULL,
+ NULL,
+ 1); /* ask for binary results */
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* divide by zero -- but server won't realize until execution; 3 params */
+ paramValues[0] = (char *) &binaryIntVal;
+ paramLengths[0] = sizeof(binaryIntVal);
+ paramFormats[0] = 1; /* binary */
+ paramValues[1] = "2";
+ paramLengths[1] = strlen(paramValues[1]) + 1;
+ paramFormats[1] = 0; /* text */
+ paramValues[2] = (char *) "everyone's little $$ in \"dollar\"";
+ paramLengths[2] = strlen(paramValues[2]) + 1;
+ paramFormats[2] = 0; /* text */
+ res = PQexecParams(
+ conn,
+ "SELECT 1 / (random() / 2)::int + $1::int + $2::int, $3::text /*exec*/",
+ 3, /* # params */
+ NULL, /* let the backend deduce param type */
+ paramValues,
+ paramLengths,
+ paramFormats,
+ 1 /* ask for binary results */
+ );
+
+ if (PQresultStatus(res) != PGRES_FATAL_ERROR)
+ {
+ fprintf(stderr, "SELECT succeeded but was supposed to fail\n");
+ PQclear(res);
+ exit_nicely(conn);
+ }
+ PQclear(res);
+ printf("Division by zero on execution expected, "
+ "got an error message from server: %s\n"
+ "Please make sure it has been logged "
+ "with bind parameters in server log\n",
+ PQerrorMessage(conn));
+
+ /* close the connection to the database and cleanup */
+ PQfinish(conn);
+
+ return 0;
+}
--
2.20.1
On 2019-Dec-06, Pavel Stehule wrote:
p� 6. 12. 2019 v 18:57 odes�latel Alvaro Herrera <alvherre@2ndquadrant.com>
napsal:
(It seems a bit weird that if I assign NULL to :two pgbench stores the
empty string, instead of the NULL that I get as in the above which is
what happens when the variable is not defined at all. That's probably a
bug in \gset ...)psql session variables cannot to hold NULL
I also just learned that the :foo expansion in pgbench behaves crazily
when in normal query mode, particularly re. quotes in variables.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
I added some tests to the pgbench suite in the attached. I also finally
put it the business to limit the length of parameters reported.
I'd probably drop testlibpq5.c, since it seems a bit pointless now ...
I took a look through the v17 patches.
0001:
The two header comments for appendStringInfoStringQuoted really ought to
define the maxlen parameter, rather than making readers reverse-engineer
what that does.
I'm not very sold on the enlargeStringInfo call in the maxlen>0 code path,
mainly because its calculation of the needed space seems entirely wrong:
(a) it's considering maxlen not the actual length, and (b) on the other
hand it's not accounting for quotes and ellipses. Forcing an inadequate
enlargement is probably worse than doing nothing, so I'd be inclined to
just drop that.
It is a very bad idea that this is truncating text without regard to
multibyte character boundaries.
0002:
Seems like BuildParamLogString's "valueLen" parameter ought to be called
"maxlen", for consistency with 0001 and because "valueLen" is at best
misleading about what the parameter means.
I'd toss the enlargeStringInfo call here too, as it seems overly
complicated and underly correct or useful.
Probably doing MemoryContextReset after each parameter (even the last one!)
is a net loss compared to just letting it go till the end. Although
I'd be inclined to use ALLOCSET_DEFAULT_SIZES not SMALL_SIZES if you
do it like that.
Please do not throw away the existing comment "/* Free result of encoding
conversion, if any */" in exec_bind_message.
As above, this risks generating partial multibyte characters. You might
be able to get away with letting appendStringInfoStringQuoted do the
multibyte-aware truncation, but you probably have to copy more than just
one more extra byte first.
I have zero faith in this:
+ params_errcxt.arg = (void *) &((ParamsErrorCbData)
+ { portal->name, params });
How do you know where the compiler is putting that value, ie what
its lifespan is going to be? Better to use an explicit variable.
I concur with dropping testlibpq5.
regards, tom lane
On 2019-Dec-07, Tom Lane wrote:
0001:
It is a very bad idea that this is truncating text without regard to
multibyte character boundaries.
Hmm. So it turns out that we recently made stringinfo.c exported to
frontends in libpgcommon. Clipping strings at a given character size
means we need some facility like pg_encoding_mbcliplen.
However, 1) pgcommon does not have wchar.c (libpq does have it), and 2)
mbcliplen is not in wchar.c anyway.
I see four possible ways forward, with nuances. In order of preference:
1. change enough of the build system so that pg_encoding_mbcliplen is
available. (Offhand I see no reason why we couldn't move the
function from mbutils.c to wchar.c, but I haven't tried.)
2. in frontend, ignore the specified length; always copy the whole
string.
3. don't allow length clipping in frontend. (We have no use for it
in the near future.)
3a. Throw a run-time error if it's requested.
3b. throw a compile-time error if requested
4. don't provide this new function #if FRONTEND
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2019-Dec-09, Alvaro Herrera wrote:
I see four possible ways forward, with nuances. In order of preference:
1. change enough of the build system so that pg_encoding_mbcliplen is
available. (Offhand I see no reason why we couldn't move the
function from mbutils.c to wchar.c, but I haven't tried.)
Ah, this seems to require putting encnames.o and wchar.o in src/common's
Makefile and moving pg_encoding_mbcliplen() from mbutils.c to wchar.c.
Fairly simple.
(There's a little bit of additional fallout because libpq also has
encnames/wchar in its roster, but I think this means we can just remove
them from there and let them be acquired via libpgcommon.)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
On 2019-Dec-07, Tom Lane wrote:
It is a very bad idea that this is truncating text without regard to
multibyte character boundaries.
I see four possible ways forward, with nuances. In order of preference:
1. change enough of the build system so that pg_encoding_mbcliplen is
available. (Offhand I see no reason why we couldn't move the
function from mbutils.c to wchar.c, but I haven't tried.)
I'd be in favor of this if it doesn't turn out to require nasty
contortions. My gut feeling (like yours, without having looked) is that
the incremental amount of code to be moved into wchar.c wouldn't be much.
BTW, not really the fault of this patch, but I wonder if we shouldn't
make an effort to push the FRONTEND-available parts of wchar.c into
src/common, just to make the system structure and build rules less
confusing. I think the current situation only exists because this
code predates our invention of src/common.
regards, tom lane
I wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
1. change enough of the build system so that pg_encoding_mbcliplen is
available. (Offhand I see no reason why we couldn't move the
function from mbutils.c to wchar.c, but I haven't tried.)
I'd be in favor of this if it doesn't turn out to require nasty
contortions. My gut feeling (like yours, without having looked) is that
the incremental amount of code to be moved into wchar.c wouldn't be much.
Actually, on second thought --- the issue here is not so much how much new
code shows up in libpgcommon, and more that executables using stringinfo.o
will now find themselves pulling in all of wchar.o, where before they
might've pulled in none of it. We need to look at how much code bloat
ensues. In the end it might be smart to put this new functionality in
some separate source file instead of dropping it into stringinfo.c.
(It could also be that all the executables that need stringinfo.o are
already pulling in wchar functionality for other reasons, but we should
check the code-size implications before assuming this is fine.)
regards, tom lane
On 2019-Dec-09, Tom Lane wrote:
I wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
1. change enough of the build system so that pg_encoding_mbcliplen is
available. (Offhand I see no reason why we couldn't move the
function from mbutils.c to wchar.c, but I haven't tried.)I'd be in favor of this if it doesn't turn out to require nasty
contortions. My gut feeling (like yours, without having looked) is that
the incremental amount of code to be moved into wchar.c wouldn't be much.Actually, on second thought --- the issue here is not so much how much new
code shows up in libpgcommon, and more that executables using stringinfo.o
will now find themselves pulling in all of wchar.o, where before they
might've pulled in none of it. We need to look at how much code bloat
ensues. In the end it might be smart to put this new functionality in
some separate source file instead of dropping it into stringinfo.c.
(It could also be that all the executables that need stringinfo.o are
already pulling in wchar functionality for other reasons, but we should
check the code-size implications before assuming this is fine.)
Hmm, that's a good thought. On the one hand, nothing in the frontend
needs this new function ... the only current use is pg_waldump which
likely doesn't need to quote anything, and I just looked at Andres'
programmatically- written node support code which is why we have
StringInfo in frontend in the first place -- it also doesn't care.
So rather than mess with stringinfo.c at all I could just create
stringinfo_server.c and put this function there, compiled only for
backend ...
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
So rather than mess with stringinfo.c at all I could just create
stringinfo_server.c and put this function there, compiled only for
backend ...
Good point: if we make a separate source file then we don't have
to solve any of the code-movement issues till somebody wants this
functionality in frontend. But we should expect that that day might
come eventually, so I don't much like "stringinfo_server.c" as the
file name. It'll look pretty silly once we start compiling it for
frontend. Perhaps "appendquoted.c" or some such?
regards, tom lane
On 2019-Dec-09, Tom Lane wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
So rather than mess with stringinfo.c at all I could just create
stringinfo_server.c and put this function there, compiled only for
backend ...Good point: if we make a separate source file then we don't have
to solve any of the code-movement issues till somebody wants this
functionality in frontend. But we should expect that that day might
come eventually, so I don't much like "stringinfo_server.c" as the
file name. It'll look pretty silly once we start compiling it for
frontend. Perhaps "appendquoted.c" or some such?
Okay, so here are two patches. I had already used the name
stringinfo_mb.c, so that's what's v19. (I'm fine with renaming it to
appendquoted.c but we might gain other such functions in the future.
Other opinions?)
The other patch (v18) puts the new function together with moving wchar
to pgcommon.
The API is different in each case: if we want it available in frontend,
we need to pass the encoding as a parameter rather than use
GetDatabaseEncoding().
This is pg_waldump with the first patch (src/utils/mb/stringinfo_mb.c):
$ nm --print-size /pgsql/install/master/bin/pg_waldump | grep -i stringinfo
000000000000a8f0 0000000000000060 T appendBinaryStringInfo
000000000000a980 0000000000000051 T appendBinaryStringInfoNT
000000000000a770 00000000000000d7 T appendStringInfo
000000000000a850 0000000000000050 T appendStringInfoChar
000000000000a8a0 000000000000004d T appendStringInfoSpaces
000000000000a950 0000000000000027 T appendStringInfoString
000000000000a630 0000000000000083 T appendStringInfoVA
000000000000a6c0 00000000000000af T enlargeStringInfo
000000000000a5e0 000000000000002b T initStringInfo
000000000000a5a0 0000000000000038 T makeStringInfo
000000000000a610 0000000000000015 T resetStringInfo
$ ls -l /pgsql/install/master/bin/pg_waldump
-rwxr-xr-x 1 alvherre alvherre 647576 dic 9 16:23 /pgsql/install/master/bin/pg_waldump*
This is with v18:
$ nm --print-size /pgsql/install/master/bin/pg_waldump | grep -i stringinfo
000000000000c8f0 0000000000000060 T appendBinaryStringInfo
000000000000c980 0000000000000051 T appendBinaryStringInfoNT
000000000000c770 00000000000000d7 T appendStringInfo
000000000000c850 0000000000000050 T appendStringInfoChar
000000000000c8a0 000000000000004d T appendStringInfoSpaces
000000000000c950 0000000000000027 T appendStringInfoString
000000000000c9e0 00000000000001b1 T appendStringInfoStringQuoted
000000000000c630 0000000000000083 T appendStringInfoVA
000000000000c6c0 00000000000000af T enlargeStringInfo
000000000000c5e0 000000000000002b T initStringInfo
000000000000c5a0 0000000000000038 T makeStringInfo
000000000000c610 0000000000000015 T resetStringInfo
$ ls -l /pgsql/install/master/bin/pg_waldump
-rwxr-xr-x 1 alvherre alvherre 704808 dic 9 16:29 /pgsql/install/master/bin/pg_waldump*
While the function itself is tiny (though it's the largest of all
stringinfo functions, hmm), it has led to a rather huge bloating of the
binary. That's because the new binary contains a number of functions
from wchar.c, like you said, such as
$ nm --print-size /pgsql/install/master/bin/pg_waldump | grep dsplen
000000000000d820 0000000000000026 t pg_ascii_dsplen
000000000000e200 0000000000000005 t pg_big5_dsplen
000000000000eb30 0000000000000030 T pg_encoding_dsplen
000000000000dac0 000000000000002e t pg_euccn_dsplen
000000000000d960 0000000000000036 t pg_eucjp_dsplen
000000000000d9b0 000000000000002e t pg_euckr_dsplen
000000000000dc10 000000000000002e t pg_euctw_dsplen
000000000000e230 0000000000000005 t pg_gb18030_dsplen
000000000000e210 0000000000000005 t pg_gbk_dsplen
000000000000dd10 0000000000000005 t pg_johab_dsplen
000000000000e170 0000000000000026 t pg_latin1_dsplen
000000000000e0f0 0000000000000034 t pg_mule_dsplen
000000000000e1c0 0000000000000036 t pg_sjis_dsplen
000000000000e220 0000000000000005 t pg_uhc_dsplen
000000000000e870 000000000000013e t pg_utf_dsplen
and some others.
All in all, it seems like v19 is the way to go.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v18-0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 36f0dc29c1187c49c5072d48796ccbb3369ed8f7 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v18] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
This version of the patch makes the new function available to frontends,
and puts wchar.c/encnames.c in libpgport.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 11 +----
src/backend/utils/mb/mbutils.c | 31 ------------
src/backend/utils/mb/wchar.c | 47 ++++++++++++++++++
src/common/Makefile | 9 +++-
src/common/stringinfo.c | 71 +++++++++++++++++++++++++++
src/include/lib/stringinfo.h | 10 ++++
src/pl/plpgsql/src/pl_exec.c | 39 +++++----------
src/test/regress/expected/plpgsql.out | 14 ++++++
src/test/regress/sql/plpgsql.sql | 13 +++++
9 files changed, 178 insertions(+), 67 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..a5e398b2f5 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,8 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, GetDatabaseEncoding(),
+ pstring, 0);
pfree(pstring);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 6b08b77717..34e9fd9f2c 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -865,37 +865,6 @@ pg_mbcliplen(const char *mbstr, int len, int limit)
len, limit);
}
-/*
- * pg_mbcliplen with specified encoding
- */
-int
-pg_encoding_mbcliplen(int encoding, const char *mbstr,
- int len, int limit)
-{
- mblen_converter mblen_fn;
- int clen = 0;
- int l;
-
- /* optimization for single byte encoding */
- if (pg_encoding_max_length(encoding) == 1)
- return cliplen(mbstr, len, limit);
-
- mblen_fn = pg_wchar_table[encoding].mblen;
-
- while (len > 0 && *mbstr)
- {
- l = (*mblen_fn) ((const unsigned char *) mbstr);
- if ((clen + l) > limit)
- break;
- clen += l;
- if (clen == limit)
- break;
- len -= l;
- mbstr += l;
- }
- return clen;
-}
-
/*
* Similar to pg_mbcliplen except the limit parameter specifies the
* character length, not the byte length.
diff --git a/src/backend/utils/mb/wchar.c b/src/backend/utils/mb/wchar.c
index b2d598cbee..f0e57593f7 100644
--- a/src/backend/utils/mb/wchar.c
+++ b/src/backend/utils/mb/wchar.c
@@ -14,6 +14,10 @@
#include "mb/pg_wchar.h"
+/* Internal functions */
+static int cliplen(const char *str, int len, int limit);
+
+
/*
* Operations on multi-byte encodings are driven by a table of helper
* functions.
@@ -1861,6 +1865,49 @@ pg_encoding_verifymb(int encoding, const char *mbstr, int len)
pg_wchar_table[PG_SQL_ASCII].mbverify((const unsigned char *) mbstr, len));
}
+/* mbcliplen for any single-byte encoding */
+static int
+cliplen(const char *str, int len, int limit)
+{
+ int l = 0;
+
+ len = Min(len, limit);
+ while (l < len && str[l])
+ l++;
+ return l;
+}
+
+/*
+ * pg_mbcliplen with specified encoding
+ */
+int
+pg_encoding_mbcliplen(int encoding, const char *mbstr,
+ int len, int limit)
+{
+ mblen_converter mblen_fn;
+ int clen = 0;
+ int l;
+
+ /* optimization for single byte encoding */
+ if (pg_encoding_max_length(encoding) == 1)
+ return cliplen(mbstr, len, limit);
+
+ mblen_fn = pg_wchar_table[encoding].mblen;
+
+ while (len > 0 && *mbstr)
+ {
+ l = (*mblen_fn) ((const unsigned char *) mbstr);
+ if ((clen + l) > limit)
+ break;
+ clen += l;
+ if (clen == limit)
+ break;
+ len -= l;
+ mbstr += l;
+ }
+ return clen;
+}
+
/*
* fetch maximum length of a given encoding
*/
diff --git a/src/common/Makefile b/src/common/Makefile
index ffb0f6edff..0146918758 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -78,6 +78,8 @@ else
OBJS_COMMON += sha2.o
endif
+backend_src = $(top_srcdir)/src/backend
+
# A few files are currently only built for frontend, not server
# (Mkvcbuild.pm has a copy of this list, too)
OBJS_FRONTEND = \
@@ -85,7 +87,9 @@ OBJS_FRONTEND = \
fe_memutils.o \
file_utils.o \
logging.o \
- restricted_token.o
+ restricted_token.o \
+ wchar.o \
+ encnames.o
# foo.o, foo_shlib.o, and foo_srv.o are all built from foo.c
OBJS_SHLIB = $(OBJS_FRONTEND:%.o=%_shlib.o)
@@ -158,6 +162,9 @@ kwlist_d.h: $(top_srcdir)/src/include/parser/kwlist.h $(GEN_KEYWORDLIST_DEPS)
# that you don't get broken parsing code, even in a non-enable-depend build.
keywords.o keywords_shlib.o keywords_srv.o: kwlist_d.h
+encnames.c wchar.c: % : $(backend_src)/utils/mb/%
+ rm -f $@ && $(LN_S) $< .
+
# The code imported from Ryu gets a pass on declaration-after-statement,
# in order to keep it more closely aligned with its upstream.
RYU_FILES = d2s.o f2s.o
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..5198ef97ac 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -30,6 +30,7 @@
#endif
#include "lib/stringinfo.h"
+#include "mb/pg_wchar.h"
/*
@@ -178,6 +179,76 @@ appendStringInfoString(StringInfo str, const char *s)
appendBinaryStringInfo(str, s, strlen(s));
}
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append up to maxlen characters from s (which is in the given encoding) to
+ * str, or the whole input string if maxlen <= 0, adding single quotes around
+ * it and doubling all single quotes. Add an ellipsis if the copy is
+ * incomplete.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, int encoding, const char *s,
+ int maxlen)
+{
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
+ *chunk_end;
+ bool ellipsis;
+ int slen;
+
+ Assert(str != NULL);
+
+ slen = strlen(s);
+ if (maxlen > 0 && maxlen < slen)
+ {
+ int finallen = pg_encoding_mbcliplen(encoding, s, slen, maxlen);
+
+ copy = pnstrdup(s, finallen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = true;
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
+
+ /* ensure sufficient space for terminators */
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ }
+
+ if (copy)
+ pfree(copy);
+}
+
/*
* appendStringInfoChar
*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..4a2af9e50b 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -113,6 +113,16 @@ extern int appendStringInfoVA(StringInfo str, const char *fmt, va_list args) pg_
*/
extern void appendStringInfoString(StringInfo str, const char *s);
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append up to maxlen characters from s (which is in the given encoding) to
+ * str, or the whole input string if maxlen <= 0, adding single quotes around
+ * it and doubling all single quotes. Add an ellipsis if the copy is
+ * incomplete.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str, int encoding,
+ const char *s, int maxlen);
+
/*------------------------
* appendStringInfoChar
* Append a single byte to str.
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..1ba1783490 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -28,6 +28,7 @@
#include "executor/spi.h"
#include "executor/spi_priv.h"
#include "funcapi.h"
+#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "optimizer/optimizer.h"
@@ -8611,19 +8612,12 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ GetDatabaseEncoding(),
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid),
+ 0);
paramno++;
}
@@ -8661,19 +8655,12 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ GetDatabaseEncoding(),
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]),
+ 0);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..cd2c79f4d5 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que mirase bien lo que hacia?'''
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
v19-0001-Add-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 817041a46a31ed951d84b6039bd469feea577052 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v19] Add appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
Put the new function in a new backend-only file, stringinfo_mb.c.
Author: Alexey Bashtanov and Ãlvaro Herrera, after a suggestion from Andres Freund
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--
src/backend/utils/mb/Makefile | 1 +
src/backend/utils/mb/stringinfo_mb.c | 87 +++++++++++++++++++++++++++
src/common/stringinfo.c | 1 +
src/include/lib/stringinfo.h | 14 +++++
src/pl/plpgsql/src/pl_exec.c | 36 +++--------
src/test/regress/expected/plpgsql.out | 14 +++++
src/test/regress/sql/plpgsql.sql | 13 ++++
8 files changed, 141 insertions(+), 35 deletions(-)
create mode 100644 src/backend/utils/mb/stringinfo_mb.c
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..3d3172e83d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring, 0);
pfree(pstring);
}
diff --git a/src/backend/utils/mb/Makefile b/src/backend/utils/mb/Makefile
index 18dd758cfe..cd4a016449 100644
--- a/src/backend/utils/mb/Makefile
+++ b/src/backend/utils/mb/Makefile
@@ -16,6 +16,7 @@ OBJS = \
conv.o \
encnames.o \
mbutils.o \
+ stringinfo_mb.o \
wchar.o \
wstrcmp.o \
wstrncmp.o
diff --git a/src/backend/utils/mb/stringinfo_mb.c b/src/backend/utils/mb/stringinfo_mb.c
new file mode 100644
index 0000000000..92771431e4
--- /dev/null
+++ b/src/backend/utils/mb/stringinfo_mb.c
@@ -0,0 +1,87 @@
+/*-------------------------------------------------------------------------
+ *
+ * stringinfo_mb.c
+ *
+ * multibyte-aware additional StringInfo facilites
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/backend/utils/stringinfo_mb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "mb/pg_wchar.h"
+#include "utils/memutils.h"
+
+
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append a null-terminated string to str, adding single quotes around it
+ * and doubling all single quotes.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s, int maxlen)
+{
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
+ *chunk_end;
+ bool ellipsis;
+ int slen;
+
+ Assert(str != NULL);
+
+ slen = strlen(s);
+ if (maxlen > 0 && maxlen < slen)
+ {
+ int finallen = pg_mbcliplen(s, slen, maxlen);
+
+ copy = pnstrdup(s, finallen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = true;
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
+
+ /* ensure sufficient space for terminators */
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ }
+
+ if (copy)
+ pfree(copy);
+}
diff --git a/src/common/stringinfo.c b/src/common/stringinfo.c
index a50e587da9..90970b54b7 100644
--- a/src/common/stringinfo.c
+++ b/src/common/stringinfo.c
@@ -30,6 +30,7 @@
#endif
#include "lib/stringinfo.h"
+#include "mb/pg_wchar.h"
/*
diff --git a/src/include/lib/stringinfo.h b/src/include/lib/stringinfo.h
index e27942728e..aadd00b2fb 100644
--- a/src/include/lib/stringinfo.h
+++ b/src/include/lib/stringinfo.h
@@ -158,4 +158,18 @@ extern void appendBinaryStringInfoNT(StringInfo str,
*/
extern void enlargeStringInfo(StringInfo str, int needed);
+/* In stringinfo_mb.c */
+#ifndef FRONTEND
+/*------------------------
+ * appendStringInfoStringQuoted
+ * Append up to maxlen characters from s (which is in the given encoding) to
+ * str, or the whole input string if maxlen <= 0, adding single quotes around
+ * it and doubling all single quotes. Add an ellipsis if the copy is
+ * incomplete.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str,
+ const char *s, int maxlen);
+#endif
+
+
#endif /* STRINGINFO_H */
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..c3ae409f39 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,11 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid),
+ 0);
paramno++;
}
@@ -8661,19 +8653,11 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]),
+ 0);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..cd2c79f4d5 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que mirase bien lo que hacia?'''
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
Also:
* Thoughts from Andres, who was busily messing about with stringinfo.c
on his patch series for programmatic out/read node functions?
* Thoughts on changing the current usage of the ...Quoted() function in
postgres.c and pl_exec.c so that they print N characters (64) instead
of the current default of printing everything? I'm up for changing,
but have got no +1s on that. I think bloating log files with enormous
parameters is pointless.
* v18 and v19 now alwys do a "strlen(s)", i.e. they scan the whole input
string -- pointless when maxlen is given. We could avoid that for
very large input strings by doing strnlen(maxlen + MAX_MULTIBYTE_CHAR_LEN)
so that we capture our input string plus one additional multibyte
char.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
On 2019-Dec-09, Tom Lane wrote:
Good point: if we make a separate source file then we don't have
to solve any of the code-movement issues till somebody wants this
functionality in frontend. But we should expect that that day might
come eventually, so I don't much like "stringinfo_server.c" as the
file name. It'll look pretty silly once we start compiling it for
frontend. Perhaps "appendquoted.c" or some such?
Okay, so here are two patches. I had already used the name
stringinfo_mb.c, so that's what's v19. (I'm fine with renaming it to
appendquoted.c but we might gain other such functions in the future.
Other opinions?)
"stringinfo_mb.c" is fine with me.
regards, tom lane
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
* Thoughts on changing the current usage of the ...Quoted() function in
postgres.c and pl_exec.c so that they print N characters (64) instead
of the current default of printing everything? I'm up for changing,
but have got no +1s on that. I think bloating log files with enormous
parameters is pointless.
Meh ... people will inevitably complain that they needed to see the
whole value, and we'll end up having to add another configuration
variable. Let's not go there just yet.
regards, tom lane
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
Also:
* v18 and v19 now alwys do a "strlen(s)", i.e. they scan the whole input
string -- pointless when maxlen is given. We could avoid that for
very large input strings by doing strnlen(maxlen + MAX_MULTIBYTE_CHAR_LEN)
so that we capture our input string plus one additional multibyte
char.
BTW, as far as that goes, it seems to me this patch is already suffering
from a lot of premature micro-optimization. Is there even any evidence
to justify that complicated chunk-at-a-time copying strategy, rather than
doing quote-doubling the same way we do it everywhere else? The fact that
that effectively scans the input string twice means that it's not an
ironclad win compared to the naive way, and it seems like it could lose
significantly for a case with lots of quote marks. Moreover, for the
lengths of strings I expect we're mostly dealing with here, it would be
impossible to measure any savings even assuming there is some. If I were
the committer I think I'd just flush that and do it the same way as we
do it in existing code.
regards, tom lane
Hi,
On 2019-12-09 17:05:31 -0300, Alvaro Herrera wrote:
* Thoughts from Andres, who was busily messing about with stringinfo.c
on his patch series for programmatic out/read node functions?
I don't immediately see a need for mb functionality there. But to me it
seems pretty clear that we're going to need mb functionality in more
frontend programs over time. The only frontend part needing stringinfo
there is the metadata generation tool, and that doesn't care about
anything like this. The node functions themselves are obviously still
backend only.
I'm not even sure that it's worth caring over the binary size this way,
tbh. If somebody is actually concerned about that, they ought to use LTO
+ -Os, leading to unreferenced functions (and even unreferenced *parts*
of functions) getting completely thrown out.
I'd also assume that size number you cited earlier contains debug
information etc? Because after stripping / without debug information, I
get like 110k/97k for assert/optimize builds of waldump from master?
130k/113k for v18. And 110k/97k for v19.
Greetings,
Andres Freund
On 2019-Dec-09, Tom Lane wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
Also:
* v18 and v19 now alwys do a "strlen(s)", i.e. they scan the whole input
string -- pointless when maxlen is given. We could avoid that for
very large input strings by doing strnlen(maxlen + MAX_MULTIBYTE_CHAR_LEN)
so that we capture our input string plus one additional multibyte
char.BTW, as far as that goes, it seems to me this patch is already suffering
from a lot of premature micro-optimization. Is there even any evidence
to justify that complicated chunk-at-a-time copying strategy, rather than
doing quote-doubling the same way we do it everywhere else? The fact that
that effectively scans the input string twice means that it's not an
ironclad win compared to the naive way, and it seems like it could lose
significantly for a case with lots of quote marks. Moreover, for the
lengths of strings I expect we're mostly dealing with here, it would be
impossible to measure any savings even assuming there is some. If I were
the committer I think I'd just flush that and do it the same way as we
do it in existing code.
It's been us committers (Andres and I) that have derailed this patch
with some apparently overcomplicated logic. There was nothing of that
stuff in the original. See Andres review comments
/messages/by-id/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
There's also a comment nearby about growing the stringinfo to a
reasonable guess of the final size, rather than letting it grow
normally, to avoid multiple reallocs. That's why we had all that stuff
there. He has looked at execution profiles much more than I have, so I
take his word from it.
The stuff about truncating to N chars was of my own devising. If we
don't want truncation in log_parameters_on_error either, we could do
away with the stringinfo changes altogether. (I stand by my opinion
that we should truncate, but if there are contrary votes I'm happy to
stand down and just get the suggested feature pushed.)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
The API is different in each case: if we want it available in frontend,
we need to pass the encoding as a parameter rather than use
GetDatabaseEncoding().
Hm, yeah, so that's another reason we aren't going to be sharing this
code exactly with frontend. Given that, dropping it under utils/mb
seems fine for now. (I'm still wondering how much of utils/mb could
be pushed over to src/common, but that seems like an independent
project.)
Some quick review of v19:
* If it's going into utils/mb, then you should update utils/mb/README.
(Or else we should get rid of that file's explanation of all the .c
files in its directory.)
* The file header comment for stringinfo_mb.c should perhaps explain
why we didn't just drop this into stringinfo.c, along the lines
of "This is separate from stringinfo.c so that frontend users of that
file need not pull in a bunch of multibyte encoding support."
* The file header comment for stringinfo_mb.c gives its path incorrectly.
* What's the need for including utils/memutils.h here?
* Still no documentation of maxlen in the .c file, and the docs added
to the .h file are confused (what "given encoding"? also it should say
"bytes" not "characters").
* This patch shouldn't be touching stringinfo.c at all anymore.
* I'm kind of dubious that stringinfo.h is where to put the extern
either; with where we're at now, perhaps pg_wchar.h is better.
regards, tom lane
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
The stuff about truncating to N chars was of my own devising. If we
don't want truncation in log_parameters_on_error either, we could do
away with the stringinfo changes altogether. (I stand by my opinion
that we should truncate, but if there are contrary votes I'm happy to
stand down and just get the suggested feature pushed.)
I think it's a reasonable thing to do, modulo my fear that people
are going to insist on having a way to configure the limit.
regards, tom lane
On Mon, 9 Dec 2019 at 15:17, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Meh ... people will inevitably complain that they needed to see the
whole value, and we'll end up having to add another configuration
variable. Let's not go there just yet.
I haven't been following this whole thread but my initial reaction is
that this particular configuration parameter would actually carry it's
weight.
While some people have applications where they know the expected size
of the data and can safely log the entire data to the logs other
people deal with user-supplied data that can be very large. It would
suck to have something like Gitlab dump entire merge diffs to the log
and those aren't even very large, probably under a megabyte. Some
users will have individual data that are up to 1GB....
--
greg
Greg Stark <stark@mit.edu> writes:
On Mon, 9 Dec 2019 at 15:17, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Meh ... people will inevitably complain that they needed to see the
whole value, and we'll end up having to add another configuration
variable. Let's not go there just yet.
I haven't been following this whole thread but my initial reaction is
that this particular configuration parameter would actually carry it's
weight.
Possibly so. I was mainly objecting to changing existing behaviors
without offering any configuration recourse for getting back the
existing behavior.
Although it would be sensible to allow logging of parameter values
to be controlled by a new GUC, it's less clear to me that the same GUC
ought to control what plpgsql's language feature print_strict_params
does. So there would be room for discussion about that even if we
agreed on making this configurable.
regards, tom lane
On 2019-Dec-09, Tom Lane wrote:
Some quick review of v19:
Here's v20 with all these comments hopefully addressed.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v20-0001-Add-backend-only-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From 62277fcd3f63ae68495300c98f77c7e4b4713614 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 3 Dec 2019 10:08:35 -0300
Subject: [PATCH v20] Add backend-only appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Simplifies some coding that prints parameters, as well as optimize to do
it per non-quote chunks instead of per byte.
This lives separetely from common/stringinfo.c so that frontend users of
that file need not pull in unnecessary multibyte-encoding support code.
Author: Ãlvaro Herrera and Alexey Bashtanov, after a suggestion from Andres Freund
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 10 +--
src/backend/utils/mb/Makefile | 1 +
src/backend/utils/mb/README | 1 +
src/backend/utils/mb/stringinfo_mb.c | 92 +++++++++++++++++++++++++++
src/pl/plpgsql/src/pl_exec.c | 36 +++--------
src/test/regress/expected/plpgsql.out | 14 ++++
src/test/regress/sql/plpgsql.sql | 13 ++++
7 files changed, 132 insertions(+), 35 deletions(-)
create mode 100644 src/backend/utils/mb/stringinfo_mb.c
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..3d3172e83d 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -2348,7 +2348,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2363,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring, 0);
pfree(pstring);
}
diff --git a/src/backend/utils/mb/Makefile b/src/backend/utils/mb/Makefile
index 18dd758cfe..cd4a016449 100644
--- a/src/backend/utils/mb/Makefile
+++ b/src/backend/utils/mb/Makefile
@@ -16,6 +16,7 @@ OBJS = \
conv.o \
encnames.o \
mbutils.o \
+ stringinfo_mb.o \
wchar.o \
wstrcmp.o \
wstrncmp.o
diff --git a/src/backend/utils/mb/README b/src/backend/utils/mb/README
index c9bc6e6f8d..7495ca5db2 100644
--- a/src/backend/utils/mb/README
+++ b/src/backend/utils/mb/README
@@ -9,6 +9,7 @@ wchar.c: mostly static functions and a public table for mb string and
multibyte conversion
mbutils.c: public functions for the backend only.
requires conv.c and wchar.c
+stringinfo_mb.c: public backend-only multibyte-aware stringinfo functions
wstrcmp.c: strcmp for mb
wstrncmp.c: strncmp for mb
win866.c: a tool to generate KOI8 <--> CP866 conversion table
diff --git a/src/backend/utils/mb/stringinfo_mb.c b/src/backend/utils/mb/stringinfo_mb.c
new file mode 100644
index 0000000000..baef1aa39a
--- /dev/null
+++ b/src/backend/utils/mb/stringinfo_mb.c
@@ -0,0 +1,92 @@
+/*-------------------------------------------------------------------------
+ *
+ * stringinfo_mb.c
+ * Multibyte encoding-aware additional StringInfo facilites
+ *
+ * This is separate from common/stringinfo.c so that frontend users
+ * of that file need not pull in unnecessary multibyte-encoding support
+ * code.
+ *
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mb/stringinfo_mb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "mb/pg_wchar.h"
+
+
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append up to maxlen characters from s to str, or the whole input string if
+ * maxlen <= 0, adding single quotes around it and doubling all single quotes.
+ * Add an ellipsis if the copy is incomplete.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s, int maxlen)
+{
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
+ *chunk_end;
+ bool ellipsis;
+ int slen;
+
+ Assert(str != NULL);
+
+ slen = strlen(s);
+ if (maxlen > 0 && maxlen < slen)
+ {
+ int finallen = pg_mbcliplen(s, slen, maxlen);
+
+ copy = pnstrdup(s, finallen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = true;
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
+
+ /* ensure sufficient space for terminators */
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ }
+
+ if (copy)
+ pfree(copy);
+}
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..c3ae409f39 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -8611,19 +8611,11 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid),
+ 0);
paramno++;
}
@@ -8661,19 +8653,11 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]),
+ 0);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..cd2c79f4d5 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que mirase bien lo que hacia?'''
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
On 2019-Dec-10, Alvaro Herrera wrote:
On 2019-Dec-09, Tom Lane wrote:
Some quick review of v19:
Here's v20 with all these comments hopefully addressed.
Grr, I had forgotten to git add the stringinfo.h -> pg_wchar.h changes,
so the prototype isn't anywhere in v20. However, after looking at it
again, I'm not very happy with how that turned out, because pg_wchar.h
is a frontend-exposed header. So I instead propose to put it in a
separate file src/include/mb/stringinfo_mb.h.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v21-0001-Add-backend-only-appendStringInfoStringQuoted.patchtext/x-diff; charset=iso-8859-1Download
From bd8df7f46556eb8a5de5e64f04b29f4d7a174d0e Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 10 Dec 2019 14:40:56 -0300
Subject: [PATCH v21] Add backend-only appendStringInfoStringQuoted
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This provides a mechanism to emit literal values in informative
messages, such as query parameters. The new code is more complex than
what it replaces, primarily because it wants to be more efficient.
It also has the (currently unused) additional optional capability of
specifying a maximum size to print.
This lives separetely from common/stringinfo.c so that frontend users of
that file need not pull in unnecessary multibyte-encoding support code.
Author: Ãlvaro Herrera and Alexey Bashtanov, after a suggestion from Andres Freund
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/20190920203905.xkv5udsd5dxfs6tr@alap3.anarazel.de
---
src/backend/tcop/postgres.c | 11 +---
src/backend/utils/mb/Makefile | 1 +
src/backend/utils/mb/README | 1 +
src/backend/utils/mb/stringinfo_mb.c | 92 +++++++++++++++++++++++++++
src/include/mb/stringinfo_mb.h | 24 +++++++
src/pl/plpgsql/src/pl_exec.c | 37 ++++-------
src/test/regress/expected/plpgsql.out | 14 ++++
src/test/regress/sql/plpgsql.sql | 13 ++++
8 files changed, 158 insertions(+), 35 deletions(-)
create mode 100644 src/backend/utils/mb/stringinfo_mb.c
create mode 100644 src/include/mb/stringinfo_mb.h
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 3b85e48333..512209a38c 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -48,6 +48,7 @@
#include "libpq/pqformat.h"
#include "libpq/pqsignal.h"
#include "mb/pg_wchar.h"
+#include "mb/stringinfo_mb.h"
#include "miscadmin.h"
#include "nodes/print.h"
#include "optimizer/optimizer.h"
@@ -2348,7 +2349,6 @@ errdetail_params(ParamListInfo params)
Oid typoutput;
bool typisvarlena;
char *pstring;
- char *p;
appendStringInfo(¶m_str, "%s$%d = ",
paramno > 0 ? ", " : "",
@@ -2364,14 +2364,7 @@ errdetail_params(ParamListInfo params)
pstring = OidOutputFunctionCall(typoutput, prm->value);
- appendStringInfoCharMacro(¶m_str, '\'');
- for (p = pstring; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶m_str, *p);
- appendStringInfoCharMacro(¶m_str, *p);
- }
- appendStringInfoCharMacro(¶m_str, '\'');
+ appendStringInfoStringQuoted(¶m_str, pstring, 0);
pfree(pstring);
}
diff --git a/src/backend/utils/mb/Makefile b/src/backend/utils/mb/Makefile
index 18dd758cfe..cd4a016449 100644
--- a/src/backend/utils/mb/Makefile
+++ b/src/backend/utils/mb/Makefile
@@ -16,6 +16,7 @@ OBJS = \
conv.o \
encnames.o \
mbutils.o \
+ stringinfo_mb.o \
wchar.o \
wstrcmp.o \
wstrncmp.o
diff --git a/src/backend/utils/mb/README b/src/backend/utils/mb/README
index c9bc6e6f8d..7495ca5db2 100644
--- a/src/backend/utils/mb/README
+++ b/src/backend/utils/mb/README
@@ -9,6 +9,7 @@ wchar.c: mostly static functions and a public table for mb string and
multibyte conversion
mbutils.c: public functions for the backend only.
requires conv.c and wchar.c
+stringinfo_mb.c: public backend-only multibyte-aware stringinfo functions
wstrcmp.c: strcmp for mb
wstrncmp.c: strncmp for mb
win866.c: a tool to generate KOI8 <--> CP866 conversion table
diff --git a/src/backend/utils/mb/stringinfo_mb.c b/src/backend/utils/mb/stringinfo_mb.c
new file mode 100644
index 0000000000..e84b9ad36f
--- /dev/null
+++ b/src/backend/utils/mb/stringinfo_mb.c
@@ -0,0 +1,92 @@
+/*-------------------------------------------------------------------------
+ *
+ * stringinfo_mb.c
+ * Multibyte encoding-aware additional StringInfo facilites
+ *
+ * This is separate from common/stringinfo.c so that frontend users
+ * of that file need not pull in unnecessary multibyte-encoding support
+ * code.
+ *
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/utils/mb/stringinfo_mb.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "mb/stringinfo_mb.h"
+#include "mb/pg_wchar.h"
+
+
+/*
+ * appendStringInfoStringQuoted
+ *
+ * Append up to maxlen characters from s to str, or the whole input string if
+ * maxlen <= 0, adding single quotes around it and doubling all single quotes.
+ * Add an ellipsis if the copy is incomplete.
+ */
+void
+appendStringInfoStringQuoted(StringInfo str, const char *s, int maxlen)
+{
+ char *copy = NULL;
+ const char *chunk_search_start,
+ *chunk_copy_start,
+ *chunk_end;
+ bool ellipsis;
+ int slen;
+
+ Assert(str != NULL);
+
+ slen = strlen(s);
+ if (maxlen > 0 && maxlen < slen)
+ {
+ int finallen = pg_mbcliplen(s, slen, maxlen);
+
+ copy = pnstrdup(s, finallen);
+ chunk_search_start = copy;
+ chunk_copy_start = copy;
+
+ ellipsis = true;
+ }
+ else
+ {
+ chunk_search_start = s;
+ chunk_copy_start = s;
+
+ ellipsis = false;
+ }
+
+ appendStringInfoCharMacro(str, '\'');
+
+ while ((chunk_end = strchr(chunk_search_start, '\'')) != NULL)
+ {
+ /* copy including the found delimiting ' */
+ appendBinaryStringInfoNT(str,
+ chunk_copy_start,
+ chunk_end - chunk_copy_start + 1);
+
+ /* in order to double it, include this ' into the next chunk as well */
+ chunk_copy_start = chunk_end;
+ chunk_search_start = chunk_end + 1;
+ }
+
+ /* copy the last chunk and terminate */
+ if (ellipsis)
+ appendStringInfo(str, "%s...'", chunk_copy_start);
+ else
+ {
+ int len = strlen(chunk_copy_start);
+
+ /* ensure sufficient space for terminators */
+ appendBinaryStringInfoNT(str, chunk_copy_start, len);
+ appendStringInfoCharMacro(str, '\'');
+ }
+
+ if (copy)
+ pfree(copy);
+}
diff --git a/src/include/mb/stringinfo_mb.h b/src/include/mb/stringinfo_mb.h
new file mode 100644
index 0000000000..e90ba8bc51
--- /dev/null
+++ b/src/include/mb/stringinfo_mb.h
@@ -0,0 +1,24 @@
+/*-------------------------------------------------------------------------
+ *
+ * stringinfo_mb.h
+ * multibyte support for StringInfo
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/mb/stringinfo_mb.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef STRINGINFO_MB_H
+#define STRINGINFO_MB_H
+
+
+#include "lib/stringinfo.h"
+
+/*
+ * Multibyte-aware StringInfo support function.
+ */
+extern void appendStringInfoStringQuoted(StringInfo str,
+ const char *s, int maxlen);
+
+#endif /* STRINGINFO_MB_H */
diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c
index 4f0de7a811..2deb7c0b12 100644
--- a/src/pl/plpgsql/src/pl_exec.c
+++ b/src/pl/plpgsql/src/pl_exec.c
@@ -28,6 +28,7 @@
#include "executor/spi.h"
#include "executor/spi_priv.h"
#include "funcapi.h"
+#include "mb/stringinfo_mb.h"
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "optimizer/optimizer.h"
@@ -8611,19 +8612,11 @@ format_expr_params(PLpgSQL_execstate *estate,
if (paramisnull)
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, paramdatum, paramtypeid);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ paramdatum,
+ paramtypeid),
+ 0);
paramno++;
}
@@ -8661,19 +8654,11 @@ format_preparedparamsdata(PLpgSQL_execstate *estate,
if (ppd->nulls[paramno] == 'n')
appendStringInfoString(¶mstr, "NULL");
else
- {
- char *value = convert_value_to_string(estate, ppd->values[paramno], ppd->types[paramno]);
- char *p;
-
- appendStringInfoCharMacro(¶mstr, '\'');
- for (p = value; *p; p++)
- {
- if (*p == '\'') /* double single quotes */
- appendStringInfoCharMacro(¶mstr, *p);
- appendStringInfoCharMacro(¶mstr, *p);
- }
- appendStringInfoCharMacro(¶mstr, '\'');
- }
+ appendStringInfoStringQuoted(¶mstr,
+ convert_value_to_string(estate,
+ ppd->values[paramno],
+ ppd->types[paramno]),
+ 0);
}
MemoryContextSwitchTo(oldcontext);
diff --git a/src/test/regress/expected/plpgsql.out b/src/test/regress/expected/plpgsql.out
index e85b29455e..cd2c79f4d5 100644
--- a/src/test/regress/expected/plpgsql.out
+++ b/src/test/regress/expected/plpgsql.out
@@ -2656,6 +2656,20 @@ create or replace function stricttest() returns void as $$
declare
x record;
p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+select stricttest();
+ERROR: query returned no rows
+DETAIL: parameters: p1 = '2', p3 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que mirase bien lo que hacia?'''
+CONTEXT: PL/pgSQL function stricttest() line 8 at SQL statement
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
p3 text := 'foo';
begin
-- too many rows
diff --git a/src/test/regress/sql/plpgsql.sql b/src/test/regress/sql/plpgsql.sql
index 70deadfbea..d841d8c0f9 100644
--- a/src/test/regress/sql/plpgsql.sql
+++ b/src/test/regress/sql/plpgsql.sql
@@ -2280,6 +2280,19 @@ end$$ language plpgsql;
select stricttest();
+create or replace function stricttest() returns void as $$
+declare
+x record;
+p1 int := 2;
+p3 text := $a$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$a$;
+begin
+ -- no rows
+ select * from foo where f1 = p1 and f1::text = p3 into strict x;
+ raise notice 'x.f1 = %, x.f2 = %', x.f1, x.f2;
+end$$ language plpgsql;
+
+select stricttest();
+
create or replace function stricttest() returns void as $$
declare
x record;
--
2.20.1
On 2019-Dec-07, Tom Lane wrote:
0002:
Here's a version of this part with fixes for these comments. It applies
on top of the stringinfo_mb.c patch sent elsewhere in the thread.
(If we were to add a "log_parameters_on_error_max_length" GUC to decide
the length to log, we would get rid of the remaining magical numbers in
this code).
Seems like BuildParamLogString's "valueLen" parameter ought to be called
"maxlen", for consistency with 0001 and because "valueLen" is at best
misleading about what the parameter means.I'd toss the enlargeStringInfo call here too, as it seems overly
complicated and underly correct or useful.Probably doing MemoryContextReset after each parameter (even the last one!)
is a net loss compared to just letting it go till the end. Although
I'd be inclined to use ALLOCSET_DEFAULT_SIZES not SMALL_SIZES if you
do it like that.Please do not throw away the existing comment "/* Free result of encoding
conversion, if any */" in exec_bind_message.As above, this risks generating partial multibyte characters. You might
be able to get away with letting appendStringInfoStringQuoted do the
multibyte-aware truncation, but you probably have to copy more than just
one more extra byte first.I have zero faith in this:
+ params_errcxt.arg = (void *) &((ParamsErrorCbData) + { portal->name, params });How do you know where the compiler is putting that value, ie what
its lifespan is going to be? Better to use an explicit variable.I concur with dropping testlibpq5.
regards, tom lane
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v21-0002-Emit-parameter-values-during-query-bind-execute-.patchtext/x-diff; charset=us-asciiDownload
From 45eca917877ea72122478fded75d5722a18d92aa Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 2 Dec 2019 16:02:48 -0300
Subject: [PATCH v21 2/2] Emit parameter values during query bind/execute
errors
---
doc/src/sgml/config.sgml | 23 ++++
src/backend/nodes/params.c | 114 +++++++++++++++++
src/backend/tcop/postgres.c | 118 +++++++++++-------
src/backend/utils/misc/guc.c | 10 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pgbench/t/001_pgbench_with_server.pl | 35 ++++++
src/include/nodes/params.h | 10 ++
src/include/utils/guc.h | 1 +
8 files changed, 265 insertions(+), 47 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 53ac14490a..5d1c90282f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6595,6 +6595,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..c3320ee1c4 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,11 +15,16 @@
#include "postgres.h"
+#include "access/xact.h"
+#include "lib/stringinfo.h"
+#include "mb/stringinfo_mb.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
+#include "utils/portal.h"
/*
@@ -44,6 +49,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->paramValuesStr = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +64,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied, either.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -158,6 +166,8 @@ EstimateParamListSpace(ParamListInfo paramLI)
* RestoreParamList can be used to recreate a ParamListInfo based on the
* serialized representation; this will be a static, self-contained copy
* just as copyParamList would create.
+ *
+ * paramValuesStr is not included.
*/
void
SerializeParamList(ParamListInfo paramLI, char **start_address)
@@ -251,3 +261,107 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString
+ * Prepare for reporting parameter values.
+ *
+ * Create a string to represent parameters list for logging and save it to
+ * params->paramValuesStr. If caller already knows textual representations for
+ * some parameters, it can pass an array of exactly params->numParams values
+ * as knownTextValues, which can contain NULLs for the individual values not
+ * known. NULL can be given if no parameters are known.
+ *
+ * If maxlen is not zero, that's the maximum number of characters of the
+ * input string printed; an ellipsis is added if more characters exist.
+ * (Added quotes are not considered.)
+ *
+ * This should only be called if log_parameters_on_error is set, or caller
+ * wants to report the query parameters for other reasons.
+ */
+void
+BuildParamLogString(ParamListInfo params, char **knownTextValues, int maxlen)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ StringInfoData buf;
+
+ /*
+ * Nothing to do if it's is already set; also, no work if the param fetch
+ * hook is in use. (Mostly because it's unimplemented ... but that's
+ * because it doesn't seem a case worth pursuing.) Also, avoid calling
+ * user-defined I/O functions when in an aborted transaction. It might
+ * be possible to improve on this when some knownTextValues exist.
+ */
+ if (params->paramValuesStr != NULL ||
+ params->paramFetch != NULL ||
+ IsAbortedTransactionBlockState())
+ return;
+
+ /* Initialize the output stringinfo, in caller's memory context */
+ initStringInfo(&buf);
+
+ /* Use a temporary context to call output functions, just in case */
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *param = ¶ms->params[paramno];
+
+ appendStringInfo(&buf,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (param->isnull || !OidIsValid(param->ptype))
+ appendStringInfoString(&buf, "NULL");
+ else
+ {
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ appendStringInfoStringQuoted(&buf, knownTextValues[paramno],
+ maxlen);
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ char *pstring;
+
+ getTypeOutputInfo(param->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, param->value);
+ appendStringInfoStringQuoted(&buf, pstring, maxlen);
+ }
+ }
+ }
+
+ params->paramValuesStr = buf.data;
+
+ MemoryContextSwitchTo(oldCxt);
+ MemoryContextDelete(tmpCxt);
+}
+
+/*
+ * ParamsErrorCallback - callback for printing parameters in error context
+ *
+ * Note that this is a no-op unless BuildParamLogString has been called
+ * beforehand.
+ */
+void
+ParamsErrorCallback(void *arg)
+{
+ ParamsErrorCbData *data = (ParamsErrorCbData *) arg;
+
+ if (data == NULL ||
+ data->params == NULL ||
+ data->params->paramValuesStr == NULL)
+ return;
+
+ if (data->portalName && data->portalName[0] != '\0')
+ errcontext("extended query \"%s\" with parameters: %s",
+ data->portalName, data->params->paramValuesStr);
+ else
+ errcontext("extended query with parameters: %s",
+ data->params->paramValuesStr);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 512209a38c..5eef1ae41c 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1614,6 +1614,8 @@ exec_bind_message(StringInfo input_message)
bool save_log_statement_stats = log_statement_stats;
bool snapshot_set = false;
char msec_str[32];
+ ParamsErrorCbData params_data;
+ ErrorContextCallback params_errcxt;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1753,6 +1755,8 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ char **knownTextValues = NULL; /* allocate on first use */
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1820,9 +1824,32 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * Free result of encoding conversion, if any, and save a copy
+ * for later when logging parameters.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ {
+ MemoryContext oldcxt;
+
+ oldcxt = MemoryContextSwitchTo(MessageContext);
+ if (knownTextValues == NULL)
+ knownTextValues =
+ palloc0(numParams * sizeof(char *));
+ /*
+ * Note: must copy at least two more full characters
+ * than BuildParamLogString wants to see; otherwise it
+ * might fail to include the ellipsis.
+ */
+ knownTextValues[paramno] =
+ pnstrdup(pstring, 64 + 2 * MAX_MULTIBYTE_CHAR_LEN);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1872,6 +1899,14 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * Prepare for printing parameters, if configured to do so. (This is
+ * saved in the portal, so that they'll appear when the query is
+ * executed later.)
+ */
+ if (log_parameters_on_error)
+ BuildParamLogString(params, knownTextValues, 64);
}
else
params = NULL;
@@ -1879,6 +1914,14 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* Set the error callback so that parameters are logged, as needed */
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_data.portalName = portal->name;
+ params_data.params = params;
+ params_errcxt.arg = (void *) ¶ms_data;
+ error_context_stack = ¶ms_errcxt;
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1924,6 +1967,12 @@ exec_bind_message(StringInfo input_message)
*/
PortalSetResultFormat(portal, numRFormats, rformats);
+ /*
+ * Done binding; remove the parameters error callback. Entries emitted
+ * later determine independently whether to log the parameters or not.
+ */
+ error_context_stack = error_context_stack->previous;
+
/*
* Send BindComplete.
*/
@@ -1980,6 +2029,8 @@ exec_execute_message(const char *portal_name, long max_rows)
bool execute_is_fetch;
bool was_logged = false;
char msec_str[32];
+ ParamsErrorCbData params_data;
+ ErrorContextCallback params_errcxt;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2104,8 +2155,16 @@ exec_execute_message(const char *portal_name, long max_rows)
CHECK_FOR_INTERRUPTS();
/*
- * Okay to run the portal.
+ * Okay to run the portal. Set the error callback so that parameters are
+ * logged. The parameters must have been saved during the bind phase.
*/
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_data.portalName = portal->name;
+ params_data.params = portalParams;
+ params_errcxt.arg = (void *) ¶ms_data;
+ error_context_stack = ¶ms_errcxt;
+
if (max_rows <= 0)
max_rows = FETCH_ALL;
@@ -2119,6 +2178,9 @@ exec_execute_message(const char *portal_name, long max_rows)
receiver->rDestroy(receiver);
+ /* Done executing; remove the params error callback */
+ error_context_stack = error_context_stack->previous;
+
if (completed)
{
if (is_xact_command)
@@ -2329,51 +2391,13 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (params && params->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
+ BuildParamLogString(params, NULL, 0);
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoStringQuoted(¶m_str, pstring, 0);
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ if (params->paramValuesStr &&
+ params->paramValuesStr[0] != '\0')
+ errdetail("parameters: %s", params->paramValuesStr);
}
return 0;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ba74bf9f7d..8d951ce404 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9541879c1f..087190ce63 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 1845869016..021d7a1826 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -270,6 +270,41 @@ COMMIT;
}
});
+# Verify server logging of parameters
+$node->append_conf('postgresql.conf', "log_parameters_on_error = true");
+$node->reload;
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: division by zero},
+ qr{CONTEXT: extended query with parameters: \$1 = '1', \$2 = NULL}
+ ],
+ 'server parameter logging',
+ {
+ '001_param_1' => q{select '1' as one \gset
+SELECT 1 / (random() / 2)::int, :one::int, :two::int;
+}
+ });
+
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: invalid input syntax for type json},
+ qr[CONTEXT: JSON data, line 1: \{ invalid\.\.\.
+extended query with parameters: \$1 = '\{ invalid ', \$2 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que ...'$]m
+ ],
+ 'server parameter logging',
+ {
+ '001_param_2' => q[select '{ invalid ' as value \gset
+select $$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$$ as long \gset
+select column1::jsonb from (values (:value), (:long)) as q;
+]
+ });
+
# test expressions
# command 1..3 and 23 depend on random seed which is used to call srandom.
pgbench(
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..d7a7e1f191 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *paramValuesStr; /* params as a single string for errors */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -149,6 +150,12 @@ typedef struct ParamExecData
bool isnull;
} ParamExecData;
+/* type of argument for ParamsErrorCallback */
+typedef struct ParamsErrorCbData
+{
+ const char *portalName;
+ ParamListInfo params;
+} ParamsErrorCbData;
/* Functions found in src/backend/nodes/params.c */
extern ParamListInfo makeParamList(int numParams);
@@ -156,5 +163,8 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern void BuildParamLogString(ParamListInfo params, char **paramTextValues,
+ int valueLen);
+extern void ParamsErrorCallback(void *arg);
#endif /* PARAMS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
--
2.20.1
On 2019-Dec-10, Alvaro Herrera wrote:
On 2019-Dec-10, Alvaro Herrera wrote:
On 2019-Dec-09, Tom Lane wrote:
Some quick review of v19:
Here's v20 with all these comments hopefully addressed.
Grr, I had forgotten to git add the stringinfo.h -> pg_wchar.h changes,
so the prototype isn't anywhere in v20. However, after looking at it
again, I'm not very happy with how that turned out, because pg_wchar.h
is a frontend-exposed header. So I instead propose to put it in a
separate file src/include/mb/stringinfo_mb.h.
Pushed 0001.
Thanks for all the reviews
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Here's a curious thing that happens with this patch. If you have
log_duration set so that parameters are logged during the bind phase,
and then an error occurs during the execution phase but you don't have
log_parameters_on_error set true, the second error will log the
parameters nonetheless ... because they were saved in the ParamListInfo
struct by the errdetail_params() call in the check_log_durations block
during bind.
I'm not sure this is a terrible problem, but does anybody think we
*not* save params->paramValuesStr across a log_duration when
log_parameters_on_error is not set?
(I think this is somewhat equivalent to the "was_logged" case in
check_log_duration itself.)
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Alexey: I would appreciate it if you give this patch a spin. Let me
know if it does what you wanted it to do.
On 2019-Dec-10, Alvaro Herrera wrote:
Here's a curious thing that happens with this patch. If you have
log_duration set so that parameters are logged during the bind phase,
and then an error occurs during the execution phase but you don't have
log_parameters_on_error set true, the second error will log the
parameters nonetheless ... because they were saved in the ParamListInfo
struct by the errdetail_params() call in the check_log_durations block
during bind.
AFAICS this has a simple solution, which is to stop saving the parameter
string in BuildParamLogString; instead, just *return* the string.
Caller can then assign it into params->paramValuesStr if appropriate
(which, in the case of errdetail_params(), it is not.) v22 does it like
that.
There is still a much smaller issue that if you have both log_durations
set to log the params during bind, and log_parameters_on_error to true,
the parameters will appear twice. But I think that's correct by
definition.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v22-0001-Emit-parameter-values-during-query-bind-execute-.patchtext/x-diff; charset=iso-8859-1Download
From 57546105624ac5184a356dc227f0820240e61b62 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Mon, 2 Dec 2019 16:02:48 -0300
Subject: [PATCH v22] Emit parameter values during query bind/execute errors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This makes such log entries more useful, since the cause of the error
can be dependent on the parameter values.
Author: Alexey Bashtanov, Ãlvaro Herrera
Discussion: https://postgr.es/m/0146a67b-a22a-0519-9082-bc29756b93a2@imap.cc
Reviewed-by: Peter Eisentraut, Andres Freund, Tom Lane
---
doc/src/sgml/config.sgml | 23 ++++
src/backend/nodes/params.c | 105 ++++++++++++++++
src/backend/tcop/postgres.c | 119 +++++++++++-------
src/backend/utils/misc/guc.c | 10 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pgbench/t/001_pgbench_with_server.pl | 35 ++++++
src/include/nodes/params.h | 10 ++
src/include/utils/guc.h | 1 +
8 files changed, 257 insertions(+), 47 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 53ac14490a..5d1c90282f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -6595,6 +6595,29 @@ log_line_prefix = '%m [%p] %q%u@%d/%a '
</listitem>
</varlistentry>
+ <varlistentry id="guc-log-parameters-on-error" xreflabel="log_parameters_on_error">
+ <term><varname>log_parameters_on_error</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>log_parameters_on_error</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Controls whether bind parameters are logged when a statement is logged
+ as a result of <xref linkend="guc-log-min-error-statement"/>.
+ It adds some overhead, as postgres will compute and store textual
+ representations of parameter values in memory for all statements,
+ even if they eventually do not get logged.
+ This setting has no effect on statements logged due to
+ <xref linkend="guc-log-min-duration-statement"/> or
+ <xref linkend="guc-log-statement"/> settings, as they are always logged
+ with parameters.
+ The default is <literal>off</literal>.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-statement" xreflabel="log_statement">
<term><varname>log_statement</varname> (<type>enum</type>)
<indexterm>
diff --git a/src/backend/nodes/params.c b/src/backend/nodes/params.c
index cf4387e40f..907da0e7cf 100644
--- a/src/backend/nodes/params.c
+++ b/src/backend/nodes/params.c
@@ -15,11 +15,14 @@
#include "postgres.h"
+#include "access/xact.h"
+#include "mb/stringinfo_mb.h"
#include "nodes/bitmapset.h"
#include "nodes/params.h"
#include "storage/shmem.h"
#include "utils/datum.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
/*
@@ -44,6 +47,7 @@ makeParamList(int numParams)
retval->paramCompileArg = NULL;
retval->parserSetup = NULL;
retval->parserSetupArg = NULL;
+ retval->paramValuesStr = NULL;
retval->numParams = numParams;
return retval;
@@ -58,6 +62,8 @@ makeParamList(int numParams)
* set of parameter values. If dynamic parameter hooks are present, we
* intentionally do not copy them into the result. Rather, we forcibly
* instantiate all available parameter values and copy the datum values.
+ *
+ * paramValuesStr is not copied, either.
*/
ParamListInfo
copyParamList(ParamListInfo from)
@@ -158,6 +164,8 @@ EstimateParamListSpace(ParamListInfo paramLI)
* RestoreParamList can be used to recreate a ParamListInfo based on the
* serialized representation; this will be a static, self-contained copy
* just as copyParamList would create.
+ *
+ * paramValuesStr is not included.
*/
void
SerializeParamList(ParamListInfo paramLI, char **start_address)
@@ -251,3 +259,100 @@ RestoreParamList(char **start_address)
return paramLI;
}
+
+/*
+ * BuildParamLogString
+ * Return a string that represent the parameter list, for logging.
+ *
+ * If caller already knows textual representations for some parameters, it can
+ * pass an array of exactly params->numParams values as knownTextValues, which
+ * can contain NULLs for the individual values not known. NULL can be given if
+ * no parameters are known.
+ *
+ * If maxlen is not zero, that's the maximum number of characters of the
+ * input string printed; an ellipsis is added if more characters exist.
+ * (Added quotes are not considered.)
+ */
+char *
+BuildParamLogString(ParamListInfo params, char **knownTextValues, int maxlen)
+{
+ MemoryContext tmpCxt,
+ oldCxt;
+ StringInfoData buf;
+
+ /*
+ * No work if the param fetch hook is in use. Also, it's not possible to
+ * do this in an aborted transaction. (It might be possible to improve on
+ * this last point when some knownTextValues exist, but it seems tricky.)
+ */
+ if (params->paramFetch != NULL ||
+ IsAbortedTransactionBlockState())
+ return NULL;
+
+ /* Initialize the output stringinfo, in caller's memory context */
+ initStringInfo(&buf);
+
+ /* Use a temporary context to call output functions, just in case */
+ tmpCxt = AllocSetContextCreate(CurrentMemoryContext,
+ "BuildParamLogString",
+ ALLOCSET_DEFAULT_SIZES);
+ oldCxt = MemoryContextSwitchTo(tmpCxt);
+
+ for (int paramno = 0; paramno < params->numParams; paramno++)
+ {
+ ParamExternData *param = ¶ms->params[paramno];
+
+ appendStringInfo(&buf,
+ "%s$%d = ",
+ paramno > 0 ? ", " : "",
+ paramno + 1);
+
+ if (param->isnull || !OidIsValid(param->ptype))
+ appendStringInfoString(&buf, "NULL");
+ else
+ {
+ if (knownTextValues != NULL && knownTextValues[paramno] != NULL)
+ appendStringInfoStringQuoted(&buf, knownTextValues[paramno],
+ maxlen);
+ else
+ {
+ Oid typoutput;
+ bool typisvarlena;
+ char *pstring;
+
+ getTypeOutputInfo(param->ptype, &typoutput, &typisvarlena);
+ pstring = OidOutputFunctionCall(typoutput, param->value);
+ appendStringInfoStringQuoted(&buf, pstring, maxlen);
+ }
+ }
+ }
+
+ MemoryContextSwitchTo(oldCxt);
+ MemoryContextDelete(tmpCxt);
+
+ return buf.data;
+}
+
+/*
+ * ParamsErrorCallback - callback for printing parameters in error context
+ *
+ * Note that this is a no-op unless BuildParamLogString has been called
+ * beforehand.
+ */
+void
+ParamsErrorCallback(void *arg)
+{
+ ParamsErrorCbData *data = (ParamsErrorCbData *) arg;
+
+ if (data == NULL ||
+ data->params == NULL ||
+ data->params->paramValuesStr == NULL)
+ return;
+
+ if (data->portalName && data->portalName[0] != '\0')
+ errcontext("extended query \"%s\" with parameters: %s",
+ data->portalName, data->params->paramValuesStr);
+ else
+ errcontext("extended query with parameters: %s",
+ data->params->paramValuesStr);
+}
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 512209a38c..0b7bc1fd03 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1614,6 +1614,8 @@ exec_bind_message(StringInfo input_message)
bool save_log_statement_stats = log_statement_stats;
bool snapshot_set = false;
char msec_str[32];
+ ParamsErrorCbData params_data;
+ ErrorContextCallback params_errcxt;
/* Get the fixed part of the message */
portal_name = pq_getmsgstring(input_message);
@@ -1753,6 +1755,8 @@ exec_bind_message(StringInfo input_message)
*/
if (numParams > 0)
{
+ char **knownTextValues = NULL; /* allocate on first use */
+
params = makeParamList(numParams);
for (int paramno = 0; paramno < numParams; paramno++)
@@ -1820,9 +1824,32 @@ exec_bind_message(StringInfo input_message)
pval = OidInputFunctionCall(typinput, pstring, typioparam, -1);
- /* Free result of encoding conversion, if any */
- if (pstring && pstring != pbuf.data)
- pfree(pstring);
+ /*
+ * Free result of encoding conversion, if any, and save a copy
+ * for later when logging parameters.
+ */
+ if (pstring)
+ {
+ if (log_parameters_on_error)
+ {
+ MemoryContext oldcxt;
+
+ oldcxt = MemoryContextSwitchTo(MessageContext);
+ if (knownTextValues == NULL)
+ knownTextValues =
+ palloc0(numParams * sizeof(char *));
+ /*
+ * Note: must copy at least two more full characters
+ * than BuildParamLogString wants to see; otherwise it
+ * might fail to include the ellipsis.
+ */
+ knownTextValues[paramno] =
+ pnstrdup(pstring, 64 + 2 * MAX_MULTIBYTE_CHAR_LEN);
+ MemoryContextSwitchTo(oldcxt);
+ }
+ if (pstring != pbuf.data)
+ pfree(pstring);
+ }
}
else if (pformat == 1) /* binary mode */
{
@@ -1872,6 +1899,15 @@ exec_bind_message(StringInfo input_message)
params->params[paramno].pflags = PARAM_FLAG_CONST;
params->params[paramno].ptype = ptype;
}
+
+ /*
+ * Once all parameters have been received, prepare for printing them in
+ * errors, if configured to do so. (This is saved in the portal, so
+ * that they'll appear when the query is executed later.)
+ */
+ if (log_parameters_on_error)
+ params->paramValuesStr =
+ BuildParamLogString(params, knownTextValues, 64);
}
else
params = NULL;
@@ -1879,6 +1915,14 @@ exec_bind_message(StringInfo input_message)
/* Done storing stuff in portal's context */
MemoryContextSwitchTo(oldContext);
+ /* Set the error callback so that parameters are logged, as needed */
+ params_data.portalName = portal->name;
+ params_data.params = params;
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) ¶ms_data;
+ error_context_stack = ¶ms_errcxt;
+
/* Get the result format codes */
numRFormats = pq_getmsgint(input_message, 2);
if (numRFormats > 0)
@@ -1924,6 +1968,12 @@ exec_bind_message(StringInfo input_message)
*/
PortalSetResultFormat(portal, numRFormats, rformats);
+ /*
+ * Done binding; remove the parameters error callback. Entries emitted
+ * later determine independently whether to log the parameters or not.
+ */
+ error_context_stack = error_context_stack->previous;
+
/*
* Send BindComplete.
*/
@@ -1980,6 +2030,8 @@ exec_execute_message(const char *portal_name, long max_rows)
bool execute_is_fetch;
bool was_logged = false;
char msec_str[32];
+ ParamsErrorCbData params_data;
+ ErrorContextCallback params_errcxt;
/* Adjust destination to tell printtup.c what to do */
dest = whereToSendOutput;
@@ -2104,8 +2156,16 @@ exec_execute_message(const char *portal_name, long max_rows)
CHECK_FOR_INTERRUPTS();
/*
- * Okay to run the portal.
+ * Okay to run the portal. Set the error callback so that parameters are
+ * logged. The parameters must have been saved during the bind phase.
*/
+ params_data.portalName = portal->name;
+ params_data.params = portalParams;
+ params_errcxt.previous = error_context_stack;
+ params_errcxt.callback = ParamsErrorCallback;
+ params_errcxt.arg = (void *) ¶ms_data;
+ error_context_stack = ¶ms_errcxt;
+
if (max_rows <= 0)
max_rows = FETCH_ALL;
@@ -2119,6 +2179,9 @@ exec_execute_message(const char *portal_name, long max_rows)
receiver->rDestroy(receiver);
+ /* Done executing; remove the params error callback */
+ error_context_stack = error_context_stack->previous;
+
if (completed)
{
if (is_xact_command)
@@ -2329,51 +2392,13 @@ errdetail_execute(List *raw_parsetree_list)
static int
errdetail_params(ParamListInfo params)
{
- /* We mustn't call user-defined I/O functions when in an aborted xact */
- if (params && params->numParams > 0 && !IsAbortedTransactionBlockState())
+ if (params && params->numParams > 0)
{
- StringInfoData param_str;
- MemoryContext oldcontext;
+ char *str;
- /* This code doesn't support dynamic param lists */
- Assert(params->paramFetch == NULL);
-
- /* Make sure any trash is generated in MessageContext */
- oldcontext = MemoryContextSwitchTo(MessageContext);
-
- initStringInfo(¶m_str);
-
- for (int paramno = 0; paramno < params->numParams; paramno++)
- {
- ParamExternData *prm = ¶ms->params[paramno];
- Oid typoutput;
- bool typisvarlena;
- char *pstring;
-
- appendStringInfo(¶m_str, "%s$%d = ",
- paramno > 0 ? ", " : "",
- paramno + 1);
-
- if (prm->isnull || !OidIsValid(prm->ptype))
- {
- appendStringInfoString(¶m_str, "NULL");
- continue;
- }
-
- getTypeOutputInfo(prm->ptype, &typoutput, &typisvarlena);
-
- pstring = OidOutputFunctionCall(typoutput, prm->value);
-
- appendStringInfoStringQuoted(¶m_str, pstring, 0);
-
- pfree(pstring);
- }
-
- errdetail("parameters: %s", param_str.data);
-
- pfree(param_str.data);
-
- MemoryContextSwitchTo(oldcontext);
+ str = BuildParamLogString(params, NULL, 0);
+ if (str && str[0] != '\0')
+ errdetail("parameters: %s", str);
}
return 0;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index ba74bf9f7d..8d951ce404 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -486,6 +486,7 @@ extern const struct config_enum_entry dynamic_shared_memory_options[];
* GUC option variables that are exported from this module
*/
bool log_duration = false;
+bool log_parameters_on_error = false;
bool Debug_print_plan = false;
bool Debug_print_parse = false;
bool Debug_print_rewritten = false;
@@ -1300,6 +1301,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"log_parameters_on_error", PGC_SUSET, LOGGING_WHAT,
+ gettext_noop("Logs bind parameters of the logged statements where possible."),
+ NULL
+ },
+ &log_parameters_on_error,
+ false,
+ NULL, NULL, NULL
+ },
{
{"debug_print_parse", PGC_USERSET, LOGGING_WHAT,
gettext_noop("Logs each query's parse tree."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9541879c1f..087190ce63 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -544,6 +544,7 @@
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
+#log_parameters_on_error = off # on error log statements with bind parameters
#log_replication_commands = off
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
diff --git a/src/bin/pgbench/t/001_pgbench_with_server.pl b/src/bin/pgbench/t/001_pgbench_with_server.pl
index 1845869016..021d7a1826 100644
--- a/src/bin/pgbench/t/001_pgbench_with_server.pl
+++ b/src/bin/pgbench/t/001_pgbench_with_server.pl
@@ -270,6 +270,41 @@ COMMIT;
}
});
+# Verify server logging of parameters
+$node->append_conf('postgresql.conf', "log_parameters_on_error = true");
+$node->reload;
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: division by zero},
+ qr{CONTEXT: extended query with parameters: \$1 = '1', \$2 = NULL}
+ ],
+ 'server parameter logging',
+ {
+ '001_param_1' => q{select '1' as one \gset
+SELECT 1 / (random() / 2)::int, :one::int, :two::int;
+}
+ });
+
+pgbench(
+ '-n -t1 -c1 -M prepared',
+ 2,
+ [],
+ [
+ qr{ERROR: invalid input syntax for type json},
+ qr[CONTEXT: JSON data, line 1: \{ invalid\.\.\.
+extended query with parameters: \$1 = '\{ invalid ', \$2 = '''Valame Dios!'' dijo Sancho; ''no le dije yo a vuestra merced que ...'$]m
+ ],
+ 'server parameter logging',
+ {
+ '001_param_2' => q[select '{ invalid ' as value \gset
+select $$'Valame Dios!' dijo Sancho; 'no le dije yo a vuestra merced que mirase bien lo que hacia?'$$ as long \gset
+select column1::jsonb from (values (:value), (:long)) as q;
+]
+ });
+
# test expressions
# command 1..3 and 23 depend on random seed which is used to call srandom.
pgbench(
diff --git a/src/include/nodes/params.h b/src/include/nodes/params.h
index fd9046619c..c6310b892f 100644
--- a/src/include/nodes/params.h
+++ b/src/include/nodes/params.h
@@ -115,6 +115,7 @@ typedef struct ParamListInfoData
void *paramCompileArg;
ParserSetupHook parserSetup; /* parser setup hook */
void *parserSetupArg;
+ char *paramValuesStr; /* params as a single string for errors */
int numParams; /* nominal/maximum # of Params represented */
/*
@@ -149,6 +150,12 @@ typedef struct ParamExecData
bool isnull;
} ParamExecData;
+/* type of argument for ParamsErrorCallback */
+typedef struct ParamsErrorCbData
+{
+ const char *portalName;
+ ParamListInfo params;
+} ParamsErrorCbData;
/* Functions found in src/backend/nodes/params.c */
extern ParamListInfo makeParamList(int numParams);
@@ -156,5 +163,8 @@ extern ParamListInfo copyParamList(ParamListInfo from);
extern Size EstimateParamListSpace(ParamListInfo paramLI);
extern void SerializeParamList(ParamListInfo paramLI, char **start_address);
extern ParamListInfo RestoreParamList(char **start_address);
+extern char *BuildParamLogString(ParamListInfo params, char **paramTextValues,
+ int valueLen);
+extern void ParamsErrorCallback(void *arg);
#endif /* PARAMS_H */
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 50098e63fe..41d5e1d14a 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -234,6 +234,7 @@ typedef enum
/* GUC vars that are actually declared in guc.c, rather than elsewhere */
extern bool log_duration;
+extern bool log_parameters_on_error;
extern bool Debug_print_plan;
extern bool Debug_print_parse;
extern bool Debug_print_rewritten;
--
2.20.1
Hi,
I see that you pushed this patch. I'm unfortunately unhappy with the
approach taken. As previously said, handling a lot of this from exec_*
is a mistake in my opinion. Pretty much the same problem exists for
parametrized query execution from other contexts, e.g. for queries
executed inside plpgsql. By putting the responsibility to manage error
contexts etc from with exec_*, we'd need to add a copy of that to pretty
much every place executing queries. This should all be in the portal
code, with an optimization for computing the parameter values from
strings inside postgres.c, when the input is text.
Greetings,
Andres Freund
Hello
On 2019-Dec-12, Andres Freund wrote:
I see that you pushed this patch.
Yeah, a version of it -- significantly different from what Alexey
submitted last.
I'm unfortunately unhappy with the
approach taken. As previously said, handling a lot of this from exec_*
is a mistake in my opinion. Pretty much the same problem exists for
parametrized query execution from other contexts, e.g. for queries
executed inside plpgsql. By putting the responsibility to manage error
contexts etc from with exec_*, we'd need to add a copy of that to pretty
much every place executing queries. This should all be in the portal
code, with an optimization for computing the parameter values from
strings inside postgres.c, when the input is text.
Hmm. I think there are two pieces of interest for this feature. One
saves a text version of the params; the other is setting the error
context that reports them. I agree that possibly it would work to have
PortalStart be in charge of the former.
However, AFAICS the error context must be set up in each place that's
executing the query; I don't think it would work to try to hack it in
SPI_cursor_open_internal, for example which is what plpgsql uses; rather
it has to be in plpgsql/pl_exec.c itself. (For example, you cannot even
put it directly in exec_dynquery_with_params -- it has to be in the
callers of that routine, and surround all code until after
exec_for_query.)
I think the current state of affairs is already a good improvement over
what was there before. Can it be improved? Sure.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services