encoding of PostgreSQL messages

Karsten Hilbert wrote:

Hi all !

How can I programmatically detect which encoding a
PostgreSQL server I am trying to connect to sends back
messages -- before I connect (so client_encoding and
the pg_settings table are flat out).

Hmm, isn't client_encoding reported in the startup packet sent by the
server, after auth?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Tue, Dec 23, 2008 at 06:45:17PM -0300, Alvaro Herrera wrote:

How can I programmatically detect which encoding a
PostgreSQL server I am trying to connect to sends back
messages -- before I connect (so client_encoding and
the pg_settings table are flat out).

Hmm, isn't client_encoding reported in the startup packet sent by the
server, after auth?

That would not quite be enough -- I am talking about
messages reported *during* auth, say

FATAL: password authentication failed for user "postgres"

or

fe_sendauth: no password supplied

both of which, in other locales, may contain non-ASCII characters.

Karsten
--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

On Dec 23, 2008, at 4:24 PM, Karsten Hilbert wrote:

How can I programmatically detect which encoding a
PostgreSQL server I am trying to connect to sends back
messages -- before I connect (so client_encoding and
the pg_settings table are flat out).

I don't think there is a way because you can't get any information
without authorizing first.

It is also unclear to me how things work in the other direction. When
authenticating, what if the user or database name have non-ascii
characters. Are they interpreted in the encoding of the server since
the client has not established an encoding?

John DeSoi, Ph.D.

Karsten Hilbert wrote:

On Tue, Dec 23, 2008 at 06:45:17PM -0300, Alvaro Herrera wrote:

Hmm, isn't client_encoding reported in the startup packet sent by the
server, after auth?

That would not quite be enough -- I am talking about
messages reported *during* auth, say

FATAL: password authentication failed for user "postgres"

or

fe_sendauth: no password supplied

both of which, in other locales, may contain non-ASCII characters.

Those are sent in the server encoding IIRC (which admittedly you don't
have a way to know, at that point.)

And I'm now wondering if we should delay initializing the translation
stuff until after client_encoding has been reported.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Mon, Dec 29, 2008 at 09:07:14AM -0300, Alvaro Herrera wrote:

That would not quite be enough -- I am talking about
messages reported *during* auth, say

FATAL: password authentication failed for user "postgres"

or

fe_sendauth: no password supplied

both of which, in other locales, may contain non-ASCII characters.

Those are sent in the server encoding IIRC (which admittedly you don't
have a way to know, at that point.)

And I'm now wondering if we should delay initializing the translation
stuff until after client_encoding has been reported.

Or else

- just don't pass those messages through gettext so they are
always in 7 bit ASCII English

- do pass them through gettext but append a 7-bit ASCII
filter so things do get passed in pseudo ASCII (this will
work for many singlebyte encodings but more often than
not for multibyte ones)

Both changes would be less intrusive than postponing the
translation.

If I had a choice I would opt for the first.

Karsten
--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Karsten Hilbert <Karsten.Hilbert@gmx.net> writes:

On Mon, Dec 29, 2008 at 09:07:14AM -0300, Alvaro Herrera wrote:

And I'm now wondering if we should delay initializing the translation
stuff until after client_encoding has been reported.

Or else

- just don't pass those messages through gettext so they are
always in 7 bit ASCII English

What's the difference? The user-visible result would be the same
AFAICS. (One or the other might be less messy internally, but I'm
not sure which offhand.)

regards, tom lane

Karsten Hilbert <Karsten.Hilbert@gmx.net> writes:

On Mon, Dec 29, 2008 at 09:07:14AM -0300, Alvaro Herrera wrote:

And I'm now wondering if we should delay initializing the translation
stuff until after client_encoding has been reported.

Or else

- just don't pass those messages through gettext so they are
always in 7 bit ASCII English

What's the difference? The user-visible result would be the same
AFAICS. (One or the other might be less messy internally, but I'm
not sure which offhand.)

That was the reason for the suggestion: perhaps less messy and surely lower impact on the existing
code as it would not mean moving code later in the initialization but rather just removing the
gettext wrappers around a few strings. No difference in the result.

The difference to my other suggestion (no translation vs. translation but then replacing
characters > 127 by, say '?' or a space) is:

I could *assume* a given encoding, namely 7 bit ASCII. Or rather I could assume
that I can display the message as "something pretty similar to what the original message said,
perhaps without umlauts and accents but still recognizable in the local language".

Now, surely, I could dig down the layers to where "my application space" receives the message
from PostgreSQL and filter there. It is, however, good to have some knowledge of the encoding
where knowledge can be had.

The concrete problem is this: I connect to PostgreSQL from Python. Let's assume PG is set to German.
If the wrong password is supplied the PG error message string contains an umlaut. This is passed to
libpq, which in turn passes it to the C part of psycopg2 which then turns this into an exception. An
exception, by default in Python, is printed to the console, which may be in any encoding incompatible
with the latin1 the PG message happens to be in. Thus, printing the PG message may or may not fail
due to Unicode de-/encoding errors.

The solution is to find the right layer to take control of the encoding but this is eventually only possible
if the encoding is *known*. Thus the plea for "7-bit-ascii English by default until the encoding *can* be
known". Going to "7-bit-ascii filter of the original by default until the encoding can be known" only
tries to preserve a bit more of the original language. I may be wrong in feasibility.

Thanks for considering,
Karsten
--
Sensationsangebot verl�ngert: GMX FreeDSL - Telefonanschluss + DSL
f�r nur 16,37 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K1308T4569a

On Wednesday 31 December 2008 18:57:29 Karsten Hilbert wrote:

The solution is to find the right layer to take control of the encoding but
this is eventually only possible if the encoding is *known*. Thus the plea
for "7-bit-ascii English by default until the encoding *can* be known".
Going to "7-bit-ascii filter of the original by default until the encoding
can be known" only tries to preserve a bit more of the original language. I
may be wrong in feasibility.

The proper fix is probably to include the client encoding in the connection
startup message.

On Wednesday 31 December 2008 18:57:29 Karsten Hilbert wrote:

The solution is to find the right layer to take control of the encoding

but

this is eventually only possible if the encoding is *known*. Thus the

plea

for "7-bit-ascii English by default until the encoding *can* be known".
Going to "7-bit-ascii filter of the original by default until the

encoding

can be known" only tries to preserve a bit more of the original

language. I

may be wrong in feasibility.

The proper fix is probably to include the client encoding in the
connection startup message.

Absolutely, either the desired client encoding being sent to the server or the current
server encoding being sent to the client.

It would, however, take way longer to trickle down into being supported by client
interfaces such as psycopg2 <evil grin>

A fixed startup encoding would not need support from those parts of the equation.

Karsten
--
Psssst! Schon vom neuen GMX MultiMessenger geh�rt? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger

Peter Eisentraut <peter_e@gmx.net> writes:

On Wednesday 31 December 2008 18:57:29 Karsten Hilbert wrote:

The solution is to find the right layer to take control of the encoding but
this is eventually only possible if the encoding is *known*. Thus the plea
for "7-bit-ascii English by default until the encoding *can* be known".
Going to "7-bit-ascii filter of the original by default until the encoding
can be known" only tries to preserve a bit more of the original language. I
may be wrong in feasibility.

The proper fix is probably to include the client encoding in the connection
startup message.

What of errors occurring before such an option could be applied?

I think that ultimately it's necessary to accept that there will be some
window during connection startup where sending plain ASCII (English)
messages is the best recourse. I'm not sure what the best way to
implement that is. On reflection though, trying to mark the individual
messages that might need that treatment doesn't seem like a winner:
there's too much possibility for mistakes, or code drift causing a
marking to become wrong; and I'm not even convinced that the very same
source-code message might not fall into both categories at different
times. So having a mode switch somehow occurring inside elog.c once
we are ready to translate seems like the right approach.

regards, tom lane

The proper fix is probably to include the client encoding in the

connection

startup message.

What of errors occurring before such an option could be applied?

I think that ultimately it's necessary to accept that there will be some
window during connection startup where sending plain ASCII (English)
messages is the best recourse. I'm not sure what the best way to
implement that is. On reflection though, trying to mark the individual
messages that might need that treatment doesn't seem like a winner:
there's too much possibility for mistakes, or code drift causing a
marking to become wrong; and I'm not even convinced that the very same
source-code message might not fall into both categories at different
times. So having a mode switch somehow occurring inside elog.c once
we are ready to translate seems like the right approach.

Hm, so maybe both Peter and Alvaro are right:

1) Setting the translation wrapper to a NOOP as early as possible.

Thus, the first messages are sent in 7-bit ASCII English.

2) Including the server encoding in the very (?) first message sent to the
client *after* which ...

3) ... proper initialization of the gettext system can happen because now
the client can know what to make of them subsequent messages.

Thereby no strings should need special marking and proper translation or
not is achieved automatically ?

It is surely acceptable to see the very first messages in English and/or
7-bit ASCII - I just want to be able to know their encoding either by definition
or by being told.

Thanks,
Karsten
--
Sensationsangebot verl�ngert: GMX FreeDSL - Telefonanschluss + DSL
f�r nur 16,37 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K1308T4569a

Hm, so maybe both Peter and Alvaro are right:

1) Setting the translation wrapper to a NOOP as early as possible.

Thus, the first messages are sent in 7-bit ASCII English.

Despite being *marked* for translation and a translation
to exist in the .po file, that is.

Karsten
--
Sensationsangebot verl�ngert: GMX FreeDSL - Telefonanschluss + DSL
f�r nur 16,37 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K1308T4569a

On Wednesday 31 December 2008 20:23:47 Tom Lane wrote:

The proper fix is probably to include the client encoding in the
connection startup message.

What of errors occurring before such an option could be applied?

Connection errors are handled by the client, which knows the client encoding.
If the setting of the client encoding would be one of the first things to be
done on the server side, you would only have a handful of possible error
conditions left (e.g., setlocale failed, out of memory). You could choose to
report those in plain ASCII or send a special error code that the client can
resolve. Although I guess no one could fault us if "could not set language"
is reported not translated. ;-)

I think that ultimately it's necessary to accept that there will be some
window during connection startup where sending plain ASCII (English)
messages is the best recourse.

Ultimately yes. But we currently handle the client encoding quite late in the
startup sequence so that many connection startup failure messages that are of
interest to normal users would likely be affected. So moving the client
encoding handling to the earliest possible phase would still be desirable.

Bruce, et al,

given the thread partially quoted below would this warrant a
TODO item "improve communication of encoding between client
and server regarding early startup messages" ?

A very usable band-aid for 8.4 - short of a proper fix -
would be the minimal-invasive sending of messages in 7-bit
English until server_encoding can be retrieved by the client
by current means.

Thanks,
Karsten

On Thu, Jan 01, 2009 at 08:33:56PM +0200, Peter Eisentraut wrote:

Subject: Re: [GENERAL] encoding of PostgreSQL messages
User-Agent: KMail/1.9.9

On Wednesday 31 December 2008 20:23:47 Tom Lane wrote:

The proper fix is probably to include the client encoding in the
connection startup message.

What of errors occurring before such an option could be applied?

Connection errors are handled by the client, which knows the client encoding.
If the setting of the client encoding would be one of the first things to be
done on the server side, you would only have a handful of possible error
conditions left (e.g., setlocale failed, out of memory). You could choose to
report those in plain ASCII or send a special error code that the client can
resolve. Although I guess no one could fault us if "could not set language"
is reported not translated. ;-)

I think that ultimately it's necessary to accept that there will be some
window during connection startup where sending plain ASCII (English)
messages is the best recourse.

Ultimately yes. But we currently handle the client encoding quite late in the
startup sequence so that many connection startup failure messages that are of
interest to normal users would likely be affected. So moving the client
encoding handling to the earliest possible phase would still be desirable.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

Added to TODO:

Improve encoding of connection startup messages sent to the client

Currently some authentication error messages are sent in the server
encoding

* http://archives.postgresql.org/pgsql-general/2008-12/msg00801.php
* http://archives.postgresql.org/pgsql-general/2009-01/msg00005.php

---------------------------------------------------------------------------

Karsten Hilbert wrote:

Bruce, et al,

given the thread partially quoted below would this warrant a
TODO item "improve communication of encoding between client
and server regarding early startup messages" ?

A very usable band-aid for 8.4 - short of a proper fix -
would be the minimal-invasive sending of messages in 7-bit
English until server_encoding can be retrieved by the client
by current means.

Thanks,
Karsten

On Thu, Jan 01, 2009 at 08:33:56PM +0200, Peter Eisentraut wrote:

Subject: Re: [GENERAL] encoding of PostgreSQL messages
User-Agent: KMail/1.9.9

On Wednesday 31 December 2008 20:23:47 Tom Lane wrote:

The proper fix is probably to include the client encoding in the
connection startup message.

What of errors occurring before such an option could be applied?

Connection errors are handled by the client, which knows the client encoding.
If the setting of the client encoding would be one of the first things to be
done on the server side, you would only have a handful of possible error
conditions left (e.g., setlocale failed, out of memory). You could choose to
report those in plain ASCII or send a special error code that the client can
resolve. Although I guess no one could fault us if "could not set language"
is reported not translated. ;-)

I think that ultimately it's necessary to accept that there will be some
window during connection startup where sending plain ASCII (English)
messages is the best recourse.

Ultimately yes. But we currently handle the client encoding quite late in the
startup sequence so that many connection startup failure messages that are of
interest to normal users would likely be affected. So moving the client
encoding handling to the earliest possible phase would still be desirable.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi,

This topic seems to be related to the bug report
[ODBC] Localized error messages, wrong charset
.

Bruce Momjian wrote:

Added to TODO:

Improve encoding of connection startup messages sent to the client

Currently some authentication error messages are sent in the server
encoding

It it true ?
IIRC the backend knows nothing about the server encoding in
authentication phase.

Psqlodbc Unicode driver sends connection startup message which
contains the client_encoding(=UTF8) guc parameter. Attached is
a trial patch so that the psqlodbc Unicode driver can get
properly localized password error messages.

regards,
Hiroshi Inoue

Can someone comment on this?

---------------------------------------------------------------------------

Hiroshi Inoue wrote:

Hi,

This topic seems to be related to the bug report
[ODBC] Localized error messages, wrong charset
.

Bruce Momjian wrote:

Added to TODO:

Improve encoding of connection startup messages sent to the client

Currently some authentication error messages are sent in the server
encoding

It it true ?
IIRC the backend knows nothing about the server encoding in
authentication phase.

Psqlodbc Unicode driver sends connection startup message which
contains the client_encoding(=UTF8) guc parameter. Attached is
a trial patch so that the psqlodbc Unicode driver can get
properly localized password error messages.

regards,
Hiroshi Inoue

Index: postmaster/postmaster.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/postmaster/postmaster.c,v
retrieving revision 1.570
diff -c -c -r1.570 postmaster.c
*** postmaster/postmaster.c	4 Jan 2009 22:19:59 -0000	1.570
--- postmaster/postmaster.c	30 Jan 2009 14:05:35 -0000
***************
*** 1552,1557 ****
--- 1552,1560 ----
pstrdup(nameptr));
port->guc_options = lappend(port->guc_options,
pstrdup(valptr));
+ 				if (stricmp(nameptr, "client_encoding") == 0 &&
+ 				    stricmp(valptr, "UTF8") == 0)
+ 					bind_textdomain_codeset(PG_TEXTDOMAIN("postgres"), "UTF-8");
}
offset = valoffset + strlen(valptr) + 1;
}

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Can someone comment on this?

Looks like a horrible hack to me. Recoding stuff to the client encoding
in the server outside the existing recoding mechanism looks pretty evil
to me. Plus, it does not address the problem of what happens to
messages sent before this, it just moves the point of "before" a bit
earlier for some special cases.

I think we have discussed more proper solutions earlier in this thread.
IMO the best approach would be for the client to include the client
encoding in the startup package.

Peter Eisentraut wrote:

Bruce Momjian wrote:

Can someone comment on this?

Looks like a horrible hack to me. Recoding stuff to the client encoding
in the server outside the existing recoding mechanism looks pretty evil
to me.

Plus, it does not address the problem of what happens to
messages sent before this, it just moves the point of "before" a bit
earlier for some special cases.

I think we have discussed more proper solutions earlier in this thread.
IMO the best approach would be for the client to include the client
encoding in the startup package.

??? My patch exactly references the client_encoding included in
the startup message.

regards,
Hiroshi Inoue