Q: limit the length of log file entries?

On 9/3/25 08:40, Albrecht Dreß wrote:

Hi all,

I wonder if it would be possible to limit the length of individual log lines being written to the log file.

Background:
I have a PL/PGsql DB function 'addval(_id text, _data bytea) RETURNS VOID'. When a broken bytea is passed as 2nd parameter, something like

albrecht@test FEHLER: ungültige hexadezimale Ziffer: »r« bei Zeichen 28
albrecht@test ANWEISUNG: SELECT FROM addval('hopp', '\xbroken')

is printed in the log file. This is fine and very helpful in this case, but I have a “friendly user” who transmitted a broken hex string of ~100 KBytes as second parameter, resulting in a log line of this size which is not really readable and when done frequently quickly fills the logs.

The log settings are, inter alia

log_parameter_max_length = 256
log_parameter_max_length_on_error = 256
log_error_verbosity = TERSE

Did you reload the server after making the above changes?

Did you take a look at output from:

https://www.postgresql.org/docs/current/view-pg-settings.html

for appropriate settings name to see if something is overriding the
settings you changed?

but seem to have not the desired effect in this case.

Did it have any effect?

I use PostgreSQL version 17.6-1.pgdg13+1 on Debian Trixie.

Thanks in advance for any help,
Albrecht.

--
Adrian Klaver
adrian.klaver@aklaver.com

Albrecht =?iso-8859-1?b?RHJl3w==?= <albrecht.dress@posteo.de> writes:

I have a PL/PGsql DB function 'addval(_id text, _data bytea) RETURNS VOID'. When a broken bytea is passed as 2nd parameter, something like

albrecht@test FEHLER: ungültige hexadezimale Ziffer: »r« bei Zeichen 28
albrecht@test ANWEISUNG: SELECT FROM addval('hopp', '\xbroken')

is printed in the log file. This is fine and very helpful in this case, but I have a “friendly user” who transmitted a broken hex string of ~100 KBytes as second parameter, resulting in a log line of this size which is not really readable and when done frequently quickly fills the logs.

There isn't any provision for limiting the length of source queries
quoted in the log. Had your user sent the bytea value as a query
parameter, then log_parameter_max_length[_on_error] would have
applied, but this looks like the value was just inline in the query.

regards, tom lane

Am 03.09.25 17:56 schrieb(en) Adrian Klaver:

Did you reload the server after making the above changes?

Sure, running “systemctl reload …”.

Did you take a look at output from:

https://www.postgresql.org/docs/current/view-pg-settings.html

for appropriate settings name to see if something is overriding the
settings you changed?

Yes:

<snip>
postgres=# select name, setting, unit from pg_settings where name like 'log_parameter_%' or name = 'log_error_verbosity';
name | setting | unit
-----------------------------------+---------+------
log_error_verbosity | terse |
log_parameter_max_length | 256 | B
log_parameter_max_length_on_error | 256 | B
(3 Zeilen)
</snip>

Thanks,
Albrecht.

Am 03.09.25 21:07 schrieb(en) Tom Lane:

There isn't any provision for limiting the length of source queries
quoted in the log.

I see, thanks for the clarification. IMHO, it would be nice to have such an option, though…

Had your user sent the bytea value as a query
parameter, then log_parameter_max_length[_on_error] would have
applied, but this looks like the value was just inline in the query.

I can confirm that the limitation is applied when I call PQexecParams() from a little c test application with the data included in the paramValues array. The overlong log lines result from queries in a Python script using the psycopg2 module – no idea how this component formats the query.

Thanks again,
Albrecht.

On 9/4/25 08:48, Albrecht Dreß wrote:

Am 03.09.25 21:07 schrieb(en) Tom Lane:

There isn't any provision for limiting the length of source queries
quoted in the log.

I see, thanks for the clarification. IMHO, it would be nice to have such an option, though…

Had your user sent the bytea value as a query
parameter, then log_parameter_max_length[_on_error] would have
applied, but this looks like the value was just inline in the query.

I can confirm that the limitation is applied when I call PQexecParams() from a little c test application with the data included in the paramValues array. The overlong log lines result from queries in a Python script using the psycopg2 module – no idea how this component formats the query.

The best explanation is found from the psycopg(3) docs:

https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html

Differences from psycopg2

"Psycopg 3 sends the query and the parameters to the server separately,
instead of merging them on the client side. Server-side binding works
for normal SELECT and data manipulation statements (INSERT, UPDATE,
DELETE), but it doesn’t work with many other statements. For instance,
it doesn’t work with SET or with NOTIFY:"

Thanks again,
Albrecht.

--
Adrian Klaver
adrian.klaver@aklaver.com

On 9/4/25 08:54, Adrian Klaver wrote:

On 9/4/25 08:48, Albrecht Dreß wrote:

Am 03.09.25 21:07 schrieb(en) Tom Lane:

There isn't any provision for limiting the length of source queries
quoted in the log.

I see, thanks for the clarification.  IMHO, it would be nice to have
such an option, though…

Had your user sent the bytea value as a query
parameter, then log_parameter_max_length[_on_error] would have
applied, but this looks like the value was just inline in the query.

I can confirm that the limitation is applied when I call
PQexecParams() from a little c test application with the data included
in the paramValues array.  The overlong log lines result from queries
in a Python script using the psycopg2 module – no idea how this
component formats the query.

The best explanation is found from the psycopg(3) docs:

https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html

Differences from psycopg2

"Psycopg 3 sends the query and the parameters to the server separately,
instead of merging them on the client side. Server-side binding works
for normal SELECT and data manipulation statements (INSERT, UPDATE,
DELETE), but it doesn’t work with many other statements. For instance,
it doesn’t work with SET or with NOTIFY:"

As example:

import psycopg2
import psycopg

conpsyc2 = psycopg2.connect("dbname=test user=postgres")
conpsyc3 = psycopg.connect("dbname=test user=postgres")

cur2 = conpsyc2.cursor()
cur3 = conpsyc3.cursor()

cur2.execute("select * from csv_test where id = %s", [1])
cur3.execute("select * from csv_test where id = %s", [1])

yields:

--cur2
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement: BEGIN
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement:
select * from csv_test where id = 1

--cur3
2025-09-04 11:18:07.158 PDT [29703] postgres@test LOG: statement: BEGIN
2025-09-04 11:18:07.159 PDT [29703] postgres@test LOG: execute
<unnamed>: select * from csv_test where id = $1
2025-09-04 11:18:07.159 PDT [29703] postgres@test DETAIL: Parameters:
$1 = '1'

Thanks again,
Albrecht.

--
Adrian Klaver
adrian.klaver@aklaver.com

Am 04.09.25 20:21 schrieb(en) Adrian Klaver:

"Psycopg 3 sends the query and the parameters to the server separately,
instead of merging them on the client side. Server-side binding works
for normal SELECT and data manipulation statements (INSERT, UPDATE,
DELETE), but it doesn’t work with many other statements. For instance,
it doesn’t work with SET or with NOTIFY:"

As example:

import psycopg2
import psycopg

conpsyc2 = psycopg2.connect("dbname=test user=postgres")
conpsyc3 = psycopg.connect("dbname=test user=postgres")

cur2 = conpsyc2.cursor()
cur3 = conpsyc3.cursor()

cur2.execute("select * from csv_test where id = %s", [1])
cur3.execute("select * from csv_test where id = %s", [1])

yields:

--cur2
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement: BEGIN
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement:
select * from csv_test where id = 1

--cur3
2025-09-04 11:18:07.158 PDT [29703] postgres@test LOG: statement: BEGIN
2025-09-04 11:18:07.159 PDT [29703] postgres@test LOG: execute
<unnamed>: select * from csv_test where id = $1
2025-09-04 11:18:07.159 PDT [29703] postgres@test DETAIL: Parameters:
$1 = '1'

Very nice example! Looks as if the (anyway broken) script should be migrated to the the newer Python module – which, looking into the docs, shouldn't be too complex…

Thanks a lot for all your helpful insight,
Albrecht.

On 9/4/25 12:43, Albrecht Dreß wrote:

Am 04.09.25 20:21 schrieb(en) Adrian Klaver:

"Psycopg 3 sends the query and the parameters to the server separately,
instead of merging them on the client side. Server-side binding works
for normal SELECT and data manipulation statements (INSERT, UPDATE,
DELETE), but it doesn’t work with many other statements. For instance,
it doesn’t work with SET or with NOTIFY:"

As example:

import psycopg2
import psycopg

conpsyc2 = psycopg2.connect("dbname=test user=postgres")
conpsyc3 = psycopg.connect("dbname=test user=postgres")

cur2 = conpsyc2.cursor()
cur3 = conpsyc3.cursor()

cur2.execute("select * from csv_test where id = %s", [1])
cur3.execute("select * from csv_test where id = %s", [1])

yields:

--cur2
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement: BEGIN
2025-09-04 11:17:30.246 PDT [29695] postgres@test LOG: statement:
select * from csv_test where id = 1

--cur3
2025-09-04 11:18:07.158 PDT [29703] postgres@test LOG: statement: BEGIN
2025-09-04 11:18:07.159 PDT [29703] postgres@test LOG: execute
<unnamed>: select * from csv_test where id = $1
2025-09-04 11:18:07.159 PDT [29703] postgres@test DETAIL: Parameters:
$1 = '1'

Very nice example! Looks as if the (anyway broken) script should be migrated to the the newer Python module – which, looking into the docs, shouldn't be too complex…

Just be sure to read:

https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html

Things that I had to deal with:

1) These:

https://www.psycopg.org/docs/extras.html#fast-execution-helpers

don't exist in psycopg(3).

Made up for, in my cases, because executemany() is faster in psycopg as
it uses pipeline mode when libpq 14+ is available.

2) COPY is not file based, so it's usage has changed. The examples here:

https://www.psycopg.org/psycopg3/docs/basic/copy.html#copy

are clear and it was not that difficult to change over.

3) with connection has changed. It now closes the connection as well as
the transaction.

FYI, you can use:

https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#client-side-binding-cursors

to keep the old client side binding style of parameters to help migrate
code over in stages.

Thanks a lot for all your helpful insight,
Albrecht.

--
Adrian Klaver
adrian.klaver@aklaver.com