BUG #5800: "corrupted" error messages (encoding problem ?)

On Tue, Dec 21, 2010 at 3:47 PM, Carlo Curatolo <genamiga@brutele.be> wrote:

The following bug has been logged online:

Bug reference:      5800
Logged by:          Carlo Curatolo
Email address:      genamiga@brutele.be
PostgreSQL version: 9.0.2 64bits
Operating system:   Windows 7 64bits
Description:        "corrupted" error messages (encoding problem ?)
Details:

On a new PC I install only Windows (French) and PosgreSQL 9.0.2 with default
parameters and without creating any object.

Example of "corrupted" error messages that occurs on several client software
(my own Java program, dbVisualizer, EMS SQL Manager)

[Error Code: 0, SQL State: 42601]  ERREUR: erreur de syntaxe � la fin de
l'entr�e

Test 1 : Windows 7 64 bits and PosgreSQL 9.0.2 64 bits

... the problem occurs

Test 2 : Windows 7 64 bits and PosgreSQL 9.0.2 32 bits

... the problem occurs

Test 3 : Windows 7 32 bits and PosgreSQL 9.0.2 32 bits

... NO problem, the error message is correct
[Error Code: 0, SQL State: 42601]  ERREUR: erreur de syntaxe à la fin de
l'entrée

This issue occurs only when PostgreSQL 9.0.2 is installed on Windows 7 64
bits...

FYI, we've been investigating this, however, whilst we can reproduce
the same issue, we see it in different circumstances which conflict
with yours:

You reported:

64 bit OS, 64 bit PG - corruption
64 bit OS, 32 bit PG - corruption
32 bit OS, 32 bit PG - OK

We see:

64 bit OS, 64 bit PG - OK
64 bit OS, 32 bit PG - corruption
32 bit OS, 32 bit PG - corruption

That implies to us that this is something environmental, rather than a
build or installer bug (between us, both installers work correctly on
their native platforms).

What does the environment look like on both of your servers? Try
running "\! set" from a psql session in each.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[Please keep messages on the mailing list]

Hi,

I don't see anything there that gives me any ideas. Anyone else have any ideas?

2011/1/7 Génération Amiga <genamiga@brutele.be>:

Hello Dave,

Here are the result of "\! set" on the servers and I still have W7-32 and W7-64 (WinXP died)
**************
W7-64 and PG9-64
**************
postgres=# \! set;
=::=::\
=C:=C:\Program Files\PostgreSQL\9.0\scripts
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\Carlo\AppData\Roaming
CLIENTENCODING_JP=0
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=WIN7-64
ComSpec=C:\Windows\system32\cmd.exe
database=postgres
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Users\Carlo
LOCALAPPDATA=C:\Users\Carlo\AppData\Local
LOGONSERVER=\\WIN7-64
NUMBER_OF_PROCESSORS=2
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PGLOCALEDIR=C:/Program Files/PostgreSQL/9.0/share/locale
PGSYSCONFDIR=C:/Program Files/PostgreSQL/9.0/etc
port=5432
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 15 Stepping 6, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=0f06
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC=C:\Users\Public
server=localhost
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Users\Carlo\AppData\Local\Temp
TMP=C:\Users\Carlo\AppData\Local\Temp
USERDOMAIN=WIN7-64
USERNAME=postgres
USERPROFILE=C:\Users\Carlo
windir=C:\Windows
postgres=#

**************
W7-32 and PG9-32
**************
postgres=# \! set;
=::=::\
=C:=C:\Program Files\PostgreSQL\9.0\scripts
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\Carlo\AppData\Roaming
CLIENTENCODING_JP=0
CommonProgramFiles=C:\Program Files\Common Files
COMPUTERNAME=WIN7-32
ComSpec=C:\Windows\system32\cmd.exe
database=postgres
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\Users\Carlo
LOCALAPPDATA=C:\Users\Carlo\AppData\Local
LOGONSERVER=\\WIN7-32
NUMBER_OF_PROCESSORS=1
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PGLOCALEDIR=C:/Program Files/PostgreSQL/9.0/share/locale
PGSYSCONFDIR=C:/Program Files/PostgreSQL/9.0/etc
port=5432
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 15 Model 2 Stepping 4, GenuineIntel
PROCESSOR_LEVEL=15
PROCESSOR_REVISION=0204
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC=C:\Users\Public
server=localhost
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Users\Carlo\AppData\Local\Temp
TMP=C:\Users\Carlo\AppData\Local\Temp
USERDOMAIN=Win7-32
USERNAME=postgres
USERPROFILE=C:\Users\Carlo
windir=C:\Windows
postgres=#
***********************************************
I can send you whatever you need, I don't touch anything to those servers. I use them only for testing and I have image backups.

Best regards.

Curatolo Carlo

= = = = = = = = Message d'origine du 2011-01-07 à 11:41:30 = = = = = = = =
On Tue, Dec 21, 2010 at 3:47 PM, Carlo Curatolo <genamiga@brutele.be> wrote:

The following bug has been logged online:

Bug reference:      5800
Logged by:          Carlo Curatolo
Email address:      genamiga@brutele.be
PostgreSQL version: 9.0.2 64bits
Operating system:   Windows 7 64bits
Description:        "corrupted" error messages (encoding problem ?)
Details:

On a new PC I install only Windows (French) and PosgreSQL 9.0.2 with default
parameters and without creating any object.

Example of "corrupted" error messages that occurs on several client software
(my own Java program, dbVisualizer, EMS SQL Manager)

[Error Code: 0, SQL State: 42601]  ERREUR: erreur de syntaxe � la fin de
l'entr�e

Test 1 : Windows 7 64 bits and PosgreSQL 9.0.2 64 bits

... the problem occurs

Test 2 : Windows 7 64 bits and PosgreSQL 9.0.2 32 bits

... the problem occurs

Test 3 : Windows 7 32 bits and PosgreSQL 9.0.2 32 bits

... NO problem, the error message is correct
[Error Code: 0, SQL State: 42601]  ERREUR: erreur de syntaxe à la fin de
l'entrée

This issue occurs only when PostgreSQL 9.0.2 is installed on Windows 7 64
bits...

FYI, we've been investigating this, however, whilst we can reproduce
the same issue, we see it in different circumstances which conflict
with yours:
You reported:
64 bit OS, 64 bit PG - corruption
64 bit OS, 32 bit PG - corruption
32 bit OS, 32 bit PG - OK
We see:
64 bit OS, 64 bit PG - OK
64 bit OS, 32 bit PG - corruption
32 bit OS, 32 bit PG - corruption
That implies to us that this is something environmental, rather than a
build or installer bug (between us, both installers work correctly on
their native platforms).
What does the environment look like on both of your servers? Try
running "\! set" from a psql session in each.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
-----
Aucun virus trouvé dans ce message.
Analyse effectuée par AVG - www.avg.fr
Version: 10.0.1191 / Base de données virale: 1435/3364 - Date: 06/01/2011

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 07.01.2011 12:35, Dave Page wrote:

[Please keep messages on the mailing list]

Hi,

I don't see anything there that gives me any ideas. Anyone else have any ideas?

Hello,

Yes.

I would like to see output of CHCP.
Which Windows codepage is used? 850?

Susanne

--
Susanne Ebrecht - 2ndQuadrant
PostgreSQL Development, 24x7 Support, Training and Services
www.2ndQuadrant.com

On Fri, Jan 7, 2011 at 12:37 PM, Susanne Ebrecht
<susanne@2ndquadrant.com> wrote:

On 07.01.2011 12:35, Dave Page wrote:

[Please keep messages on the mailing list]

Hi,

I don't see anything there that gives me any ideas. Anyone else have any
ideas?

Hello,

Yes.

I would like to see output of CHCP.
Which Windows codepage is used? 850?

In our testing, the windows codepage was 1252, and the console
codepage was 850 (giving the normal warning about the mismatch that
we've seen for years). That was the case on installations with, and
without the incorrect formatting.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

2011/1/27 Génération Amiga <genamiga@brutele.be>:

Hello Dave,

Any news about that encoding problem ?

We've been working with a couple of our friends in Japan, and it looks
like one of them has tracked down an issue in the gettext library. It
looks like we can work around it. We'l try to get it into the next
release.

Our messages don't appear on the "pgsql-bugs" web page...
How can I do that ?

Yes they do: http://archives.postgresql.org/pgsql-bugs/2010-12/msg00156.php

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

I tried with 9.0.3...same problem...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4248990.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

I tried with 9.1.alpha5...same problem...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4291142.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

On Tue, Mar 22, 2011 at 7:07 AM, genamiga <genamiga@brutele.be> wrote:

I tried with 9.0.3...same problem...

This should be resolved in 9.0.4 btw.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Just tested the 9.0.4...same problem I am affraid...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4340437.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

On Tue, Apr 26, 2011 at 10:07 AM, Carlo Curatolo <genamiga@brutele.be> wrote:

Just tested the 9.0.4...same problem I am affraid...

Uh, that's odd. I've asked someone to see if we can reproduce it again.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Last test...

On a new PC I install in this order :

- Windows 7 Pro SP1 x64 French (brand new original DVD)
- postgresql-9.0.4-1-windows_x64.exe (installing with Locale French, Belgium
or default locale, same results)
- jre-6u24-windows-x64
- dbvis_windows-x64_7_1_4.exe
- postgresql-9.0-801.jdbc4.jar for use with java test application and
DBVisualizer

Here is the Java test application source :
http://postgresql.1045698.n5.nabble.com/file/n4346044/Main.java Main.java

Here are the results :
http://postgresql.1045698.n5.nabble.com/file/n4346044/java_app_cmd.png
http://postgresql.1045698.n5.nabble.com/file/n4346044/java_app_swing.png
http://postgresql.1045698.n5.nabble.com/file/n4346044/dbvisualizer.png

Result in logfile is correct...
http://postgresql.1045698.n5.nabble.com/file/n4346044/postgresql-2011-04-28_095111.log
postgresql-2011-04-28_095111.log

Result in PGAdmin is correct...client_encoding is also UNICODE but error
message is correct like in the log file...

So...I have posted the first time on December...

I am installing my new server (database and files). I am ready to migrate
everything on it.

I have tested my application and except that everything works fine. I would
like to use PG9 in production now.

If I use that in this state, this problem can be fixed with a simple install
of a new release ?
Or do I have to reinstall everything ?
Do I have to wait a release without this problem ?

Please let me know.
Thanks in advance.

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4346044.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

On Thu, Apr 28, 2011 at 9:53 AM, Carlo Curatolo <genamiga@brutele.be> wrote:

Last test...

On a new PC I install in this order :

- Windows 7 Pro SP1 x64 French (brand new original DVD)
- postgresql-9.0.4-1-windows_x64.exe (installing with Locale French, Belgium
or default locale, same results)
- jre-6u24-windows-x64
- dbvis_windows-x64_7_1_4.exe
- postgresql-9.0-801.jdbc4.jar for use with java test application and
DBVisualizer

Here is the Java test application source :
http://postgresql.1045698.n5.nabble.com/file/n4346044/Main.java Main.java

Here are the results :
http://postgresql.1045698.n5.nabble.com/file/n4346044/java_app_cmd.png
http://postgresql.1045698.n5.nabble.com/file/n4346044/java_app_swing.png
http://postgresql.1045698.n5.nabble.com/file/n4346044/dbvisualizer.png

Result in logfile is correct...
http://postgresql.1045698.n5.nabble.com/file/n4346044/postgresql-2011-04-28_095111.log
postgresql-2011-04-28_095111.log

Result in PGAdmin is correct...client_encoding is also UNICODE but error
message is correct like in the log file...

So...I have posted the first time on December...

I am installing my new server (database and files). I am ready to migrate
everything on it.

I have tested my application and except that everything works fine. I would
like to use PG9 in production now.

If I use that in this state, this problem can be fixed with a simple install
of a new release ?
Or do I have to reinstall everything ?
Do I have to wait a release without this problem ?

So you're saying it works for you now in PostgreSQL and pgAdmin?

I can't help with Java apps or dbVisualizer I'm afraid.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

My actual PG 8.4 production server works perfectly and in the reports
correctly errors everywhere (PGAdmin, DBVisualizer, Java applications, EMS
SQL Manager Lite)...

If somebody have an idea...a workaround...a solution...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4346143.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

Same test but with the 32bits version of PG9

The problem do NOT occurs...

Everything works paerfectly everywhere...

Where I have to post my problem to have a solution ?

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4346180.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

On Thu, Apr 28, 2011 at 6:31 AM, Carlo Curatolo <genamiga@brutele.be> wrote:

Same test but with the 32bits version of PG9

The problem do NOT occurs...

Everything works paerfectly everywhere...

Where I have to post my problem to have a solution ?

The problem isn't that you are posting in the wrong place, or that the
right people aren't listening. The problem is that after repeated
attempts, we haven't been able to figure out exactly what is going
wrong here. It's not even clear to me whether this is a PostgreSQL
bug, an installer bug, or expected Windows behavior caused by some
non-obvious aspect of your configuration. The EnterpriseDB team has
made multiple attempts to reproduce this internally, and while we've
managed to create various weird behaviors, they aren't obviously the
same as what's happening to you. :-(

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Just tried with PG 9.1...same problem...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4812062.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

Just tried with PG 9.1 64bits...same problem...

--
View this message in context: http://postgresql.1045698.n5.nabble.com/BUG-5800-corrupted-error-messages-encoding-problem-tp3313951p4812066.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.

On 09/17/2011 05:10 AM, Carlo Curatolo wrote:

Just tried with PG 9.1...same problem...

Yep. There appears to be no interest in fixing this bug. All the
alternatives I proposed were rejected, and there doesn't seem to be any
concern about the issue. I'd be willing to have a go at the issue if
there some indication that a patch might be considered, but if I don't
have an approach that might even be considered for inclusion there isn't
much point.

There are several sources of differently-encoded text that may appear in
logs. These are often set to the same encoding, but need not necessarily
be. The only valid fixes are to log them to different files (with some
way to identify which encoding is used) or convert them all to a single
standard encoding - probably UTF-8 - for logging. When syslog is used,
encoding conversion is the only valid answer.

--
Craig Ringer

Craig Ringer <ringerc@ringerc.id.au> writes:

On 09/17/2011 05:10 AM, Carlo Curatolo wrote:

Just tried with PG 9.1...same problem...

Yep. There appears to be no interest in fixing this bug. All the
alternatives I proposed were rejected, and there doesn't seem to be any
concern about the issue.

The problem is to find a cure that's not worse than the disease.
I'm not exactly convinced that forcing all log messages into a common
encoding is a better behavior than allowing backends to log in their
native database encoding.

If you do want a common encoding, there's a very easy way to get it, ie,
standardize on one encoding for all your databases. People who aren't
doing that already probably have good reasons why they want to stay with
the encoding choices they've made; forcing their logs into some other
encoding isn't necessarily going to improve their lives.

... The only valid fixes are to log them to different files (with some
way to identify which encoding is used)

I don't recall having heard any serious discussion of such a design, but
perhaps doing that would satisfy some use-cases. One idea that comes to
mind is to provide a %-escape for log_filename that expands to the name
of the database encoding (or more likely, some suitable abbrevation).
The logging collector protocol would have to be expanded to include that
information, but that seems do-able.

regards, tom lane