PG 9.4.4 issue on French Windows 32 bits

On 07/08/2015 01:04 AM, Thierry Hauchard wrote:

Hy,

We can't upgrade our customers from 8.4 to 9.4.4 if they have a 32 bits
server machine (Windows 2008, 2007...).
There is no problem with PG 64 bits.

Except below you say only for majority of 64bit machines, so are the
problems when they occur the same as you are seeing on the 32bit machines?

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: s�quence d'octets invalide pour
l'encodage � UTF8 � : 0xf4 0x6c 0x65 0x20

So what is the database encoding in the 8.4 instances?

Using our application, using win_1252 client_encoding, get similar
error if we try to update record with a "\" in any string :

So what happens if you use psql and do:

\encoding win_1252

and then run the query below?

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

The same query work from PG_Admin (UTF8 encoding)

Make me think you had database in an encoding other then UTF8 previously.

Our technicien install PG cluster exactly the same way.
We have tried on 3 differents 32 bits machine, same issue
No issue on 64bits machine (majority of our customers). All works
perfectly.
We have never had any problem with PG 8.4 for long years

Is there a way to turn-over ?
Or need to wait upgrade ?

Thanks,
Thierry Hauchard
Gesteam

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

There is NO problem on 64 bits machines with PG 64 bits.
(We have not try PG 32 bits on 64 bits machine.)
There IS problem on all 32 bits machine with PG 32 bits

All databases are encoded on PG in UTF8

We set that on all connexions :
SET CLIENT_ENCODING TO WIN1252;
SET bytea_output TO escape;
SET standard_conforming_strings TO false;

I don't use psql. So i don't anderstand when i must type "\encoding
win1252".
if i use psql on the test 9.4.4 database and type "\encoding win1252",
nothing happens
If then i try a simple query updating string with "\\", value is stored
If i do same thing with some accentuated char : "\\��", there are NOT
updated in database, and no error (and nothing in PG log)
If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252;", psql return error : invalid
byte sequence for encoding "UTF8": 0xe9 0x71 0x75
If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252" (without ";"), it works

I don't see any coherent things ! :(

Thanks,
--
Thierry
Gesteam

Le 08/07/2015 15:20, Adrian Klaver a �crit :

On 07/08/2015 01:04 AM, Thierry Hauchard wrote:

Hy,

We can't upgrade our customers from 8.4 to 9.4.4 if they have a 32 bits
server machine (Windows 2008, 2007...).
There is no problem with PG 64 bits.

Except below you say only for majority of 64bit machines, so are the
problems when they occur the same as you are seeing on the 32bit
machines?

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: s�quence d'octets invalide pour
l'encodage � UTF8 � : 0xf4 0x6c 0x65 0x20

So what is the database encoding in the 8.4 instances?

Using our application, using win_1252 client_encoding, get similar
error if we try to update record with a "\" in any string :

So what happens if you use psql and do:

\encoding win_1252

and then run the query below?

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

The same query work from PG_Admin (UTF8 encoding)

Make me think you had database in an encoding other then UTF8 previously.

Our technicien install PG cluster exactly the same way.
We have tried on 3 differents 32 bits machine, same issue
No issue on 64bits machine (majority of our customers). All works
perfectly.
We have never had any problem with PG 8.4 for long years

Is there a way to turn-over ?
Or need to wait upgrade ?

Thanks,
Thierry Hauchard
Gesteam

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 07/08/2015 08:40 AM, Thierry Hauchard wrote:

There is NO problem on 64 bits machines with PG 64 bits.

I saw this line from your previous post:

"No issue on 64bits machine (majority of our customers)."

Realize now that this means the majority of your customers are using
64bit Windows. I read it as there was no problem on the majority of the
machines that ran 64bit, which implied there where some 64bit machines
that has issues. Looking from wrong end of telescope:)

(We have not try PG 32 bits on 64 bits machine.)
There IS problem on all 32 bits machine with PG 32 bits

All databases are encoded on PG in UTF8

Just to be clear the 8.4 versions of the database where set up with UTF8?

We set that on all connexions :
SET CLIENT_ENCODING TO WIN1252;
SET bytea_output TO escape;
SET standard_conforming_strings TO false;

What library are you using to make connections in your application?

I don't use psql. So i don't anderstand when i must type "\encoding
win1252".
if i use psql on the test 9.4.4 database and type "\encoding win1252",
nothing happens

Actually something does happen:

postgres@production=# \encoding
UTF8
postgres@production=# \encoding win_1252
postgres@production=# \encoding
WIN1252

For more detail see here:
http://www.postgresql.org/docs/9.4/interactive/app-psql.html

If then i try a simple query updating string with "\\", value is stored
If i do same thing with some accentuated char : "\\��", there are NOT
updated in database, and no error (and nothing in PG log)

Can you show the actual full queries and return values or errors from in
psql command line?

If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252;", psql return error : invalid
byte sequence for encoding "UTF8": 0xe9 0x71 0x75

The psql in the above confuses me. Are you referring to the command line
program psql or to the Postgres server?

Generally psql is used to refer to the command line program and Postgres
or Pg(pg) is used to refer to the server.

If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252" (without ";"), it works

I do not use pgAdmin enough, but I am guessing it adds the ; when it
parses the command.

I don't see any coherent things ! :(

Thanks,
--
Thierry
Gesteam

Le 08/07/2015 15:20, Adrian Klaver a �crit :

On 07/08/2015 01:04 AM, Thierry Hauchard wrote:

Hy,

We can't upgrade our customers from 8.4 to 9.4.4 if they have a 32 bits
server machine (Windows 2008, 2007...).
There is no problem with PG 64 bits.

Except below you say only for majority of 64bit machines, so are the
problems when they occur the same as you are seeing on the 32bit
machines?

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: s�quence d'octets invalide pour
l'encodage � UTF8 � : 0xf4 0x6c 0x65 0x20

So what is the database encoding in the 8.4 instances?

Using our application, using win_1252 client_encoding, get similar
error if we try to update record with a "\" in any string :

So what happens if you use psql and do:

\encoding win_1252

and then run the query below?

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

The same query work from PG_Admin (UTF8 encoding)

Make me think you had database in an encoding other then UTF8 previously.

Our technicien install PG cluster exactly the same way.
We have tried on 3 differents 32 bits machine, same issue
No issue on 64bits machine (majority of our customers). All works
perfectly.
We have never had any problem with PG 8.4 for long years

Is there a way to turn-over ?
Or need to wait upgrade ?

Thanks,
Thierry Hauchard
Gesteam

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Thierry Hauchard wrote:

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: séquence d'octets invalide pour
l'encodage « UTF8 » : 0xf4 0x6c 0x65 0x20

[...]

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

These sequences of bytes seem to come from LATIN1-encoded
error messages from the backend, translated to french.

0xf4 0x6c 0x65 is "ôle" which could come from "rôle"="role" in
english, a fragment of message that occurs routinely when
restoring a dump granting permissions to roles that don't exist in
the target cluster.

0xee 0x6e 0x65 is "îne" as in "chaîne" which is "string" in french. It's
plausible that the above update, given standard_conforming_strings
to false, produces the translated version of:
"nonstandard use of \\' in a string literal"
which is:
"utilisation non standard de \\' dans une chaîne littérale"
where non-surprisingly, the first non US-ASCII sequence is "îne"

See how lc_messages is configured in postgresql.conf.
Presumably it's French_France.1252 ?

If you can live with english messages, set it to C, otherwise
someone more knowledgeable in Windows might suggest a
proper explanation and fix.
Personally I don't understand in the first place how UTF-8
is handled with '*.1252' locales, as cp1252 seems
incompatible with UTF-8 by definition.

Best regards,
--
Daniel
PostgreSQL-powered mail user agent and storage: http://www.manitou-mail.org

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 07/08/2015 08:40 AM, Thierry Hauchard wrote:

There is NO problem on 64 bits machines with PG 64 bits.
(We have not try PG 32 bits on 64 bits machine.)
There IS problem on all 32 bits machine with PG 32 bits

Further thought, are the 32bit and 64bit Windows in the same version of
Windows or different?

All databases are encoded on PG in UTF8

We set that on all connexions :
SET CLIENT_ENCODING TO WIN1252;
SET bytea_output TO escape;
SET standard_conforming_strings TO false;

I don't use psql. So i don't anderstand when i must type "\encoding
win1252".
if i use psql on the test 9.4.4 database and type "\encoding win1252",
nothing happens
If then i try a simple query updating string with "\\", value is stored
If i do same thing with some accentuated char : "\\��", there are NOT
updated in database, and no error (and nothing in PG log)
If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252;", psql return error : invalid
byte sequence for encoding "UTF8": 0xe9 0x71 0x75
If i do that from PG_Admin, it works
If i type "SET CLIENT_ENCODING TO WIN1252" (without ";"), it works

I don't see any coherent things ! :(

Thanks,
--
Thierry
Gesteam

Le 08/07/2015 15:20, Adrian Klaver a �crit :

On 07/08/2015 01:04 AM, Thierry Hauchard wrote:

Hy,

We can't upgrade our customers from 8.4 to 9.4.4 if they have a 32 bits
server machine (Windows 2008, 2007...).
There is no problem with PG 64 bits.

Except below you say only for majority of 64bit machines, so are the
problems when they occur the same as you are seeing on the 32bit
machines?

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: s�quence d'octets invalide pour
l'encodage � UTF8 � : 0xf4 0x6c 0x65 0x20

So what is the database encoding in the 8.4 instances?

Using our application, using win_1252 client_encoding, get similar
error if we try to update record with a "\" in any string :

So what happens if you use psql and do:

\encoding win_1252

and then run the query below?

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

The same query work from PG_Admin (UTF8 encoding)

Make me think you had database in an encoding other then UTF8 previously.

Our technicien install PG cluster exactly the same way.
We have tried on 3 differents 32 bits machine, same issue
No issue on 64bits machine (majority of our customers). All works
perfectly.
We have never had any problem with PG 8.4 for long years

Is there a way to turn-over ?
Or need to wait upgrade ?

Thanks,
Thierry Hauchard
Gesteam

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Thierry,

Please post the output of

$ psql -l

for the database in question for both the 32-bit and 64-bit servers. That will show what encoding was specified when the databases where created.

Also, post the output of

$ psql <dbname here> <login> -c “show client_encoding”
$ psql <dbname here> <login> -c "show all" | grep lc_

s'il vous plaît. Also check if any code is setting client_encoding or any of the lc_* options on the fly.

Bob Lunney
Senior Database Engineer
AWeber Communications, LLC
1100 Manor Drive
Chalfont, PA 18914 USA

On Jul 8, 2015, at 12:48 PM, Daniel Verite <daniel@manitou-mail.org> wrote:

Thierry Hauchard wrote:

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: séquence d'octets invalide pour
l'encodage « UTF8 » : 0xf4 0x6c 0x65 0x20

[...]

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

These sequences of bytes seem to come from LATIN1-encoded
error messages from the backend, translated to french.

0xf4 0x6c 0x65 is "ôle" which could come from "rôle"="role" in
english, a fragment of message that occurs routinely when
restoring a dump granting permissions to roles that don't exist in
the target cluster.

0xee 0x6e 0x65 is "îne" as in "chaîne" which is "string" in french. It's
plausible that the above update, given standard_conforming_strings
to false, produces the translated version of:
"nonstandard use of \\' in a string literal"
which is:
"utilisation non standard de \\' dans une chaîne littérale"
where non-surprisingly, the first non US-ASCII sequence is "îne"

See how lc_messages is configured in postgresql.conf.
Presumably it's French_France.1252 ?

If you can live with english messages, set it to C, otherwise
someone more knowledgeable in Windows might suggest a
proper explanation and fix.
Personally I don't understand in the first place how UTF-8
is handled with '*.1252' locales, as cp1252 seems
incompatible with UTF-8 by definition.

Best regards,
--
Daniel
PostgreSQL-powered mail user agent and storage: http://www.manitou-mail.org

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Hi,
which build of PosgreSQL did you use?

Postgres Professional has published today its Windows PostgreSQL installer for 9.4.4, which solves some strange issues with russian encoding in psql occurring in the EDB build.
I'm not sure it can help in your case.
The installer is at http://postgrespro.ru/windows-en.html

Regards,
Ivan Panchenko

Hi,

Thanks very much Daniel ! :))
You hit the problem.

lc_messages were configured with 'French_France.1252' (as usual)
Setting it to 'C' resolve the issue.
Another way is to set client_min_messages to 'error'

So, it seems that a 32 bits PG 9.4.4 return to client the UTF8 error
generated by log translation, and not the 64 bits version or older
versions (same conf file) !

Thanks very much !

Best regards,
--
Thierry Hauchard
Gesteam

Le 08/07/2015 18:48, Daniel Verite a écrit :

Thierry Hauchard wrote:

When restoring from backup (created from 8.4 database with PG_Dump
9.4.4), the log shows errors about UTF like :
2015-07-07 17:03:35 CEST ERREUR: séquence d'octets invalide pour
l'encodage « UTF8 » : 0xf4 0x6c 0x65 0x20

[...]

UPDATE test_table SET str_field = '\\' WHERE id = 75160909
-> ERROR: invalid byte sequence for encoding "UTF8": 0xee 0x6e 0x65

These sequences of bytes seem to come from LATIN1-encoded
error messages from the backend, translated to french.

0xf4 0x6c 0x65 is "ôle" which could come from "rôle"="role" in
english, a fragment of message that occurs routinely when
restoring a dump granting permissions to roles that don't exist in
the target cluster.

0xee 0x6e 0x65 is "îne" as in "chaîne" which is "string" in french. It's
plausible that the above update, given standard_conforming_strings
to false, produces the translated version of:
"nonstandard use of \\' in a string literal"
which is:
"utilisation non standard de \\' dans une chaîne littérale"
where non-surprisingly, the first non US-ASCII sequence is "îne"

See how lc_messages is configured in postgresql.conf.
Presumably it's French_France.1252 ?

If you can live with english messages, set it to C, otherwise
someone more knowledgeable in Windows might suggest a
proper explanation and fix.
Personally I don't understand in the first place how UTF-8
is handled with '*.1252' locales, as cp1252 seems
incompatible with UTF-8 by definition.

Best regards,

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general