[PG19] psql backslash commands fail with translated messages and a non UTF-8 client encoding

On Tue, Oct 13, 2026 at 1:26 PM shihao zhong <zhong950419@gmail.com> wrote:

0001 sends the query without the label when the label is not plain
ASCII and the locale encoding differs from client_encoding. In every
other case the label is sent as before, and -E always shows it.

I'm not sure how useful it is to translate query labels. Isn't it simpler and
safer to leave them untranslated, ensuring that they contain only ASCII
characters?

The v1 approach of comparing the locale encoding with client_encoding
doesn't seem sufficient to determine whether a translated label is safe to
send. For example, with a Japanese UTF-8 locale and client_encoding=UTF8,
the patch would still send a Japanese label to a LATIN1 database, where
conversion to the database encoding would fail.

Regards,

--
Fujii Masao

Hi Fujii,

Thanks for looking into this

I'm not sure how useful it is to translate query labels. Isn't it simpler

and

safer to leave them untranslated, ensuring that they contain only ASCII
characters?

Yes, it is. v2-A does that. It drops the translation at all 84 call
sites, so a label can only be ASCII. The cost is that -E shows the
English label.

For example, with a Japanese UTF-8 locale and client_encoding=UTF8,
the patch would still send a Japanese label to a LATIN1 database, where
conversion to the database encoding would fail.

I missed that the server converts the query once more. If we want to
keep the translation, v2-B fixes v1 for this. It sends a label that is
not plain ASCII only when the locale encoding, client_encoding and
server_encoding are all the same.

Thanks,
Shihao

On Wed, Oct 14, 2026 at 3:30 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

I think that restricting things like that would provide a reasonable
compromise between safety and losing translation altogether.

Matching all three encodings may not be sufficient, since, e.g.,
OUTPUT_CHARSET can also affect the locale encoding?
I'm also not sure whether matching all three is sufficient on Windows.

where addQueryComment contains the logic to decide whether translation
is safe and then also adds the comment syntax.

Agreed. If we cannot guarantee the translation's encoding, falling back
to the original English label would preserve it in server logs and
avoid stripping comments during execution.

Regards,

--
Fujii Masao

Hi,

Looks like we agree on the three-way rule, with the API Tom suggested.

v3 attached, done the way Tom suggested. addQueryComment() uses the
translation only if it is plain ASCII, or if gettext's output
encoding, client_encoding and server_encoding are known to match.
Otherwise it uses the English label.

Matching all three encodings may not be sufficient, since, e.g.,
OUTPUT_CHARSET can also affect the locale encoding?

Yes. psql never calls bind_textdomain_codeset(), so OUTPUT_CHARSET is
the only override. With all three UTF8 and OUTPUT_CHARSET=EUC-JP,
19rc1 fails with invalid byte sequence. v3 uses English whenever
OUTPUT_CHARSET is set.

I'm also not sure whether matching all three is sufficient on Windows.

I can’t test there. psql and libintl work out the code page in different
ways on
Windows, and I could not convince myself they always agree. So on Windows
v3 uses the English label whenever the translation is not plain ASCII.

We may lose a little since the CJK code pages map to SJIS, GBK, BIG5
and UHC, which can never be server_encoding.

Thanks,
Shihao

shihao zhong <zhong950419@gmail.com> writes:

Looks like we agree on the three-way rule, with the API Tom suggested.
v3 attached, done the way Tom suggested.

It's late here, so these comments may be off-base, but:

* It occurred to me that we could avoid all the gettext_noop()
decorations by adding addQueryComment to GETTEXT_TRIGGERS.
This is a little bit magic perhaps, but we use that approach with
most functions that take a potentially-translated string.

* On Windows, this version of addQueryComment() performs a
gettext() lookup and then unconditionally discards the result.
Elsewhere it only conditionally discards the result, but neither
thing seems great. Should we invert the logic to avoid a useless
gettext lookup?

* Do we actually need to test for OUTPUT_CHARSET, given that
this code doesn't execute on Windows? My Linux man page for
gettext does not indicate that it responds to OUTPUT_CHARSET,
so I'm wondering if that is a Windows-only thing.

regards, tom lane

Hi Tom,

* It occurred to me that we could avoid all the gettext_noop()
decorations by adding addQueryComment to GETTEXT_TRIGGERS.

Done in v4. The call sites now pass a plain string. The msgids that
xgettext extracts are the same as on master.

Should we invert the logic to avoid a useless gettext lookup?

Yes. v4 checks the encodings first and calls gettext only when they
match. On Windows it never looks the label up. The one thing we lose
is a translation that happens to be plain ASCII when the encodings
differ.

* Do we actually need to test for OUTPUT_CHARSET, given that
this code doesn't execute on Windows?

Yes. glibc honors it too. On Ubuntu 24.04 with glibc 2.39,
a ja_JP.UTF-8 locale and a UTF8 database, an unpatched psql 19 works,
and with OUTPUT_CHARSET=EUC-JP it fails:

ERROR: invalid byte sequence for encoding "UTF8": 0xc2 0xd0

v4 uses the English label in that case.

Thanks,
Shihao

On Wed, Oct 14, 2026 at 1:45 PM shihao zhong <zhong950419@gmail.com> wrote:

Yes. glibc honors it too. On Ubuntu 24.04 with glibc 2.39,
a ja_JP.UTF-8 locale and a UTF8 database, an unpatched psql 19 works,
and with OUTPUT_CHARSET=EUC-JP it fails:

ERROR: invalid byte sequence for encoding "UTF8": 0xc2 0xd0

v4 uses the English label in that case.

Thanks for updating the patch!

With the v4 patch, I encountered another case that causes
an "ERROR: invalid byte sequence" error.

First, I started psql as follows:

$ OUTPUT_CHARSET=EUC-JP psql -X

When I ran \dn, the output was garbled because of the OUTPUT_CHARSET setting:

=# \dn
???????ް???
̾?? | \xBD?ͭ\xBC
--------+-------------------
public | pg_database_owner
(1 ??)

I then tried unsetting OUTPUT_CHARSET to address garbled output:

=# \setenv OUTPUT_CHARSET

Running \dn again resulted in an error:

=# \dn
ERROR: invalid byte sequence for encoding "UTF8": 0xc2 0xd0

This seems to happen because psql with the v4 patch sees the current
value of OUTPUT_CHARSET, while NLS continues to use an earlier value
(perhaps the one it saw when psql started??). As a result, seems unsetting
OUTPUT_CHARSET while psql is running can trigger this error.

We should address this case as well, or is it reasonable to ignore it as
an edge case?

Regards,

--
Fujii Masao

Hi Fujii,

We should address this case as well, or is it reasonable to ignore it as
an edge case?

I can reproduce it, and I think we should fix it. Guessing the encoding
gettext uses keeps missing cases.

v6 does two things. It treats OUTPUT_CHARSET as set if it was set when
psql started, since gettext keeps using that value.

I fuzzed it with random locales, encodings, \setenv, \encoding and \c,
12000 cases on Linux and macOS. v6 had no errors. Unpatched 19 failed in
about 4% of them.

Thanks,
Shihao

shihao zhong <zhong950419@gmail.com> writes:

We should address this case as well, or is it reasonable to ignore it as
an edge case?

I can reproduce it, and I think we should fix it. Guessing the encoding
gettext uses keeps missing cases.
v6 does two things. It treats OUTPUT_CHARSET as set if it was set when
psql started, since gettext keeps using that value.

I thought that it'd be a good idea to push the infrastructure part of
this (ie, inventing addQueryComment) separately, since that step is
bulky, boring, and different between the branches. So I did that,
and now we just need to discuss what logic to put into
addQueryComment. I've attached a v7 to show what remains to do.

I'm not convinced by the OUTPUT_CHARSET handling in v6: it only
detects the situation where OUTPUT_CHARSET is set at the moment we're
trying to output a label. If you are worried that gettext will latch
onto a no-longer-present setting then that's not really good enough.
What I put into v7 was an addition in exec_command_setenv() to
remember if we ever did \setenv OUTPUT_CHARSET. AFAIK that is the
only pathway for psql env vars to change after startup, so that seems
like it ought to be bulletproof.

Having said that, I'm really quite unconvinced that we should be
expending any effort on OUTPUT_CHARSET at all. If someone is using
that to select an encoding that doesn't match psql's locale, they
are going to have lots more problems besides this one, and it's
all self-inflicted damage. So my vote is to rip out all the logic
related to OUTPUT_CHARSET.

The other thing I'm not super pleased about is invoking
pg_get_encoding_from_locale() each time through. That seems like a
moderately expensive operation. I also wonder whether the result
can change intra-run and whether gettext will track it if so.

regards, tom lane

I wrote:

The other thing I'm not super pleased about is invoking
pg_get_encoding_from_locale() each time through. That seems like a
moderately expensive operation. I also wonder whether the result
can change intra-run and whether gettext will track it if so.

Thinking about that some more: why don't we just drop that test,
along with the OUTPUT_CHARSET hacks, and confine ourselves to
testing that (1) server_encoding equals client_encoding and
(2) the translated string passes pg_encoding_verifymbstr?

Doing that is sufficient to guarantee that the translated label
will not fail the server's encoding check, and it depends only
on behaviors that are under our control; no guessing about what
gettext will do is required. It might be that the results will
look like junk to a human inspecting the server log later, but
I doubt we could make strong promises about that anyway.

regards, tom lane

On Fri, Oct 16, 2026 at 4:34 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thinking about that some more: why don't we just drop that test,
along with the OUTPUT_CHARSET hacks, and confine ourselves to
testing that (1) server_encoding equals client_encoding and
(2) the translated string passes pg_encoding_verifymbstr?

Sounds reasonable to me.

Regards,

--
Fujii Masao

Hi,

Tom Lane <tgl@sss.pgh.pa.us> writes:

Thinking about that some more: why don't we just drop that test,
along with the OUTPUT_CHARSET hacks, and confine ourselves to
testing that (1) server_encoding equals client_encoding and
(2) the translated string passes pg_encoding_verifymbstr?

Thanks for pushing the infrastructure. v8 does what you suggest,
with one more restriction: it translates only when both encodings are
UTF8.

WIN1252 accepts byte 0x81 but has no character for it. With
LANG=ja_JP.UTF-8, psql -f against a WIN1252 database sends the
Japanese label. It runs fine, but then a UTF8 client in that database
can't read pg_stat_activity:

ERROR: character with byte sequence 0x81 in encoding "WIN1252" has no
equivalent in encoding "UTF8"

Unpatched 19 does the same.

Thanks,
Shihao

shihao zhong <zhong950419@gmail.com> writes:

Thanks for pushing the infrastructure. v8 does what you suggest,
with one more restriction: it translates only when both encodings are
UTF8.

Uh ... what?

WIN1252 accepts byte 0x81 but has no character for it.

Surely your solution is shooting the messenger. If WIN1252 behaves
that way, we need to fix its validator, not insert arbitrary
restrictions elsewhere to work around the inadequate validation.

regards, tom lane

Hi Tom,

Tom Lane <tgl@sss.pgh.pa.us> writes:

Surely your solution is shooting the messenger. If WIN1252 behaves
that way, we need to fix its validator, not insert arbitrary
restrictions elsewhere to work around the inadequate validation.

Cool. v9 does what you suggested, equal encodings plus
pg_encoding_verifymbstr.

Fixing the WIN1252 validator would be master only. I want to focus
on 19 stability until GA, and I can look at it after that.

Thanks,
Shihao

Hi Tom,

Tom Lane <tgl@sss.pgh.pa.us> writes:

Hmm, I wonder why we can't use this same logic on Windows?

I am not a windows expert and agree "guessing is harder on Windows is
no longer a factor.“

v10 drops the WIN32 exclusion.

Thanks,
Shihao

shihao zhong <zhong950419@gmail.com> writes:

Tom Lane <tgl@sss.pgh.pa.us> writes:

Hmm, I wonder why we can't use this same logic on Windows?

I am not a windows expert and agree "guessing is harder on Windows is
no longer a factor.“
v10 drops the WIN32 exclusion.

Works for me, pushed.

regards, tom lane

Works for me, pushed.

Thanks for committing it.

Surely your solution is shooting the messenger. If WIN1252 behaves
that way, we need to fix its validator, not insert arbitrary
restrictions elsewhere to work around the inadequate validation.

I had put that into my queue and planned to start work on that
after 19 GA.