unicode in 7.1

Started by Culley Harrelsonover 24 years ago24 messagesgeneral
Jump to latest
#1Culley Harrelson
culleyharrelson@yahoo.com

Hello,

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

culley

__________________________________________________
Do You Yahoo!?
Get email alerts & NEW webcam video instant messaging with Yahoo! Messenger
http://im.yahoo.com

#2Tatsuo Ishii
t-ishii@sra.co.jp
In reply to: Culley Harrelson (#1)
Re: unicode in 7.1

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Hard to tell without knowing what the configuration option was and
what kind of API you are using...
--
Tatsuo Ishii

#3Karen Ellrick
k-ellrick@sctech.co.jp
In reply to: Culley Harrelson (#1)
Re: unicode in 7.1

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Without any more info it's only a guess, but did the ISP folks forget to use
the --enable-multibyte=<favorite_charset> during the "configure" step of
installation? If they had multibyte enabled before and didn't this time,
that could explain the problem.

Regards,

--------------------------------
Karen Ellrick
S & C Technology, Inc.
1-21-35 Kusatsu-shinmachi
Hiroshima 733-0834 Japan
(from U.S. 011-81, from Japan 0) 82-293-2838
--------------------------------

#4Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Karen Ellrick (#3)
Fwd: Re: unicode in 7.1

The was corrupted in the process of the upgrade.

Is there some way to tell what the configuration options were when it was
installed? I am assuming by API you mean how am I accessing Postgres? JDBC.

Culley

X-Apparently-To: culleyharrelson@yahoo.com via web9605; 10 Sep 2001
18:19:26 -0700 (PDT)
X-Track: 1: 40
To: culleyharrelson@yahoo.com
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] unicode in 7.1
X-Mailer: Mew version 1.94.2 on Emacs 20.7 / Mule 4.1
=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
Date: Tue, 11 Sep 2001 10:19:00 +0900
From: Tatsuo Ishii <t-ishii@sra.co.jp>
X-Dispatcher: imput version 20000228(IM140)
Lines: 9

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Hard to tell without knowing what the configuration option was and
what kind of API you are using...
--
Tatsuo Ishii

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#5Tatsuo Ishii
t-ishii@sra.co.jp
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

The was corrupted in the process of the upgrade.

Is there some way to tell what the configuration options were when it was
installed?

pg_config --configure

I am assuming by API you mean how am I accessing Postgres? JDBC.

7.1's JDBC driver has been slightly enhanced from 7.0 in the sense of
encoding handling. Modify your Java applications to meet the new JDBC
driver's requirements (do not ask me how, because I'm not a Java
programmer) or stay with the old driver.
--
Tatsuo Ishii

#6Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Tatsuo Ishii (#5)
Re: Fwd: Re: unicode in 7.1

pg_config --configure produced:

--enable-locale --with-CXX --prefix=/usr --with-perl --enable-multibyte
--with-tcl --with-odbc --enable-syslog --with-python --with-openssl
--with-krb5=/usr/kerberos --sysconfdir=/etc/pgsql --mandir=/usr/share/man
--docdir=/usr/share/doc --includedir=/usr/include/pgsql
--datadir=/usr/share/pgsql

I did have some problems with JDBC not liking to open a resultset
recursively but I just altered the code. Mostly I am concerned about all
my non-english data-- I can't even recover if the database can't handle the
data.

Culley
At 01:43 PM 9/11/01 +0900, you wrote:

The was corrupted in the process of the upgrade.

Is there some way to tell what the configuration options were when it was
installed?

pg_config --configure

I am assuming by API you mean how am I accessing Postgres? JDBC.

7.1's JDBC driver has been slightly enhanced from 7.0 in the sense of
encoding handling. Modify your Java applications to meet the new JDBC
driver's requirements (do not ask me how, because I'm not a Java
programmer) or stay with the old driver.
--
Tatsuo Ishii

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#7Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Culley Harrelson (#1)
Re: unicode in 7.1

What would be the proper encoding? Essentially anything goes with the data
in this database-- I had language aid material that could potentially be
for any language you can type.

I don't think java is the issue-- from pgsql in the shell you get a
question mark ? for your most basic accented character.

culley

At 02:25 PM 9/11/01 +0200, you wrote:

On Mon, Sep 10, 2001 at 11:25:36AM -0700, Culley Harrelson wrote:

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Your database encoding is wrong. Probably SQL_ASCII. And in
7.1 the JDBC drivers takes account of database encoding and
presents invalid symbols as '?'.

--
marko

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#8Marko Kreen
markokr@gmail.com
In reply to: Culley Harrelson (#1)
Re: unicode in 7.1

On Mon, Sep 10, 2001 at 11:25:36AM -0700, Culley Harrelson wrote:

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Your database encoding is wrong. Probably SQL_ASCII. And in
7.1 the JDBC drivers takes account of database encoding and
presents invalid symbols as '?'.

--
marko

#9Marko Kreen
markokr@gmail.com
In reply to: Culley Harrelson (#7)
Re: unicode in 7.1

On Tue, Sep 11, 2001 at 05:21:22AM -0700, Culley Harrelson wrote:

What would be the proper encoding? Essentially anything goes with the data
in this database-- I had language aid material that could potentially be
for any language you can type.

UNICODE?

I don't think java is the issue-- from pgsql in the shell you get a
question mark ? for your most basic accented character.

The issue is neither PostgreSQL nor JDBC driver. The issue is
that db encoding does not match data. If you see '?' psql
then seems like the data was converted on insert to SQL_ASCII...

psql does no do any charset conversions (yet?), JDBC does.
If data is in psql OK but in JDBC wrong then you are lucky and
data is still healthy.

You should dump your database and create db with correct
encoding and reload data.

Use \l in psql to see you db-s and encodings.

--
marko

#10Barry Lind
barry@xythos.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

Culley,

With out more details of your setup, I can't give you a complete answer.
But check out the info at:

http://lab.applinet.nl/postgresql-jdbc/#CharacterEncoding

for a brief discussion of what I believe is your problem. There has
also been a number of discussions on this on the pgsql-jdbc mail list.
You might also want to check the mail archives.

thanks,
--Barry

Culley Harrelson wrote:

Show quoted text

The was corrupted in the process of the upgrade.

Is there some way to tell what the configuration options were when it
was installed? I am assuming by API you mean how am I accessing
Postgres? JDBC.

Culley

X-Apparently-To: culleyharrelson@yahoo.com via web9605; 10 Sep 2001
18:19:26 -0700 (PDT)
X-Track: 1: 40
To: culleyharrelson@yahoo.com
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] unicode in 7.1
X-Mailer: Mew version 1.94.2 on Emacs 20.7 / Mule 4.1
=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
Date: Tue, 11 Sep 2001 10:19:00 +0900
From: Tatsuo Ishii <t-ishii@sra.co.jp>
X-Dispatcher: imput version 20000228(IM140)
Lines: 9

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty

well

but I just discovered that non-english characters are now in the

database

as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Hard to tell without knowing what the configuration option was and
what kind of API you are using...
--
Tatsuo Ishii

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

#11Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#10)
Re: Fwd: Re: unicode in 7.1

Ack! I guess I am hitting this problem....

I had my database rebuilt to use UNICODE encoding. Data now appears
correctly in pgsql but not when filtered through JDBC. Unfortunately Im
using the open source DbConnectionBroker connection pooling object and I
have to dig into this to apply the fix. It is suprising to me that JDBC
has a problem with a database using UNICODE encoding?!? I obviously don't
understand the internals of this stuff <grin>

culley

At 10:06 AM 9/11/01 -0700, you wrote:

Culley,

With out more details of your setup, I can't give you a complete
answer. But check out the info at:

http://lab.applinet.nl/postgresql-jdbc/#CharacterEncoding

for a brief discussion of what I believe is your problem. There has also
been a number of discussions on this on the pgsql-jdbc mail list. You
might also want to check the mail archives.

thanks,
--Barry

Culley Harrelson wrote:

The was corrupted in the process of the upgrade.
Is there some way to tell what the configuration options were when it was
installed? I am assuming by API you mean how am I accessing Postgres? JDBC.
Culley

X-Apparently-To: culleyharrelson@yahoo.com via web9605; 10 Sep 2001
18:19:26 -0700 (PDT)
X-Track: 1: 40
To: culleyharrelson@yahoo.com
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] unicode in 7.1
X-Mailer: Mew version 1.94.2 on Emacs 20.7 / Mule 4.1
=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
Date: Tue, 11 Sep 2001 10:19:00 +0900
From: Tatsuo Ishii <t-ishii@sra.co.jp>
X-Dispatcher: imput version 20000228(IM140)
Lines: 9

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went pretty well
but I just discovered that non-english characters are now in the database
as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix this?

Hard to tell without knowing what the configuration option was and
what kind of API you are using...
--
Tatsuo Ishii

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#12Barry Lind
barry@xythos.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

Culley,

What do you get when you issue the following select statements:

select getdatabaseencoding();
select datname, encoding from pg_database;

thanks,
--Barry

Culley Harrelson wrote:

Show quoted text

Ack! I guess I am hitting this problem....

I had my database rebuilt to use UNICODE encoding. Data now appears
correctly in pgsql but not when filtered through JDBC. Unfortunately Im
using the open source DbConnectionBroker connection pooling object and I
have to dig into this to apply the fix. It is suprising to me that JDBC
has a problem with a database using UNICODE encoding?!? I obviously
don't understand the internals of this stuff <grin>

culley

At 10:06 AM 9/11/01 -0700, you wrote:

Culley,

With out more details of your setup, I can't give you a complete
answer. But check out the info at:

http://lab.applinet.nl/postgresql-jdbc/#CharacterEncoding

for a brief discussion of what I believe is your problem. There has
also been a number of discussions on this on the pgsql-jdbc mail list.
You might also want to check the mail archives.

thanks,
--Barry

Culley Harrelson wrote:

The was corrupted in the process of the upgrade.
Is there some way to tell what the configuration options were when it
was installed? I am assuming by API you mean how am I accessing
Postgres? JDBC.
Culley

X-Apparently-To: culleyharrelson@yahoo.com via web9605; 10 Sep 2001
18:19:26 -0700 (PDT)
X-Track: 1: 40
To: culleyharrelson@yahoo.com
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] unicode in 7.1
X-Mailer: Mew version 1.94.2 on Emacs 20.7 / Mule 4.1
=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
Date: Tue, 11 Sep 2001 10:19:00 +0900
From: Tatsuo Ishii <t-ishii@sra.co.jp>
X-Dispatcher: imput version 20000228(IM140)
Lines: 9

my isp recently upgraded form postgreSQL 7.0 to 7.1. It went

pretty well

but I just discovered that non-english characters are now in the

database

as a question mark-- inserting non-english characters produces a ? as
well. Any idea what has gone wrong and what we need to do to fix

this?

Hard to tell without knowing what the configuration option was and
what kind of API you are using...
--
Tatsuo Ishii

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#13Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#12)
Re: Fwd: Re: unicode in 7.1

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5

All other databases on the server had 0 for encoding.

Unfortunately my dev environment still has postgres 7.0. I updated my
connection pool object and it works on dev but not on prod with 7.1. I
really don't want to update my dev environment untill I can do the whole
machine from scratch but I fear I might have to in order to debug this...

Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#14Barry Lind
barry@xythos.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

Culley,

That all looks fine. I don't understand why you would be having
problems with these settings. Are you sure you are using the 7.1 jdbc
drivers? One other thing I can think of to check is to do a 'show
client_encoding'. This should also return UNICODE.

So to restate your problem, you get the 8bit characters returned as ?'s
through jdbc, but it works correctly through psql. Is this a correct
restatement of your problem?

thanks,
--Barry

Culley Harrelson wrote:

Show quoted text

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5

All other databases on the server had 0 for encoding.

Unfortunately my dev environment still has postgres 7.0. I updated my
connection pool object and it works on dev but not on prod with 7.1. I
really don't want to update my dev environment untill I can do the whole
machine from scratch but I fear I might have to in order to debug this...

Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#15Marko Kreen
markokr@gmail.com
In reply to: Culley Harrelson (#11)
Re: Fwd: Re: unicode in 7.1

On Tue, Sep 11, 2001 at 11:39:38AM -0700, Culley Harrelson wrote:

Ack! I guess I am hitting this problem....

I had my database rebuilt to use UNICODE encoding. Data now appears
correctly in pgsql but not when filtered through JDBC. Unfortunately Im
using the open source DbConnectionBroker connection pooling object and I
have to dig into this to apply the fix. It is suprising to me that JDBC
has a problem with a database using UNICODE encoding?!? I obviously don't
understand the internals of this stuff <grin>

With UNICODE you are claiming to JDBC 'hey, my data
is UNICODE (UTF8)'. What encoding the data fields really
are?

--
marko

#16Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#14)
Re: Fwd: Re: unicode in 7.1

No this problem was fixed when I rebuild the database with UNICODE
encoding. No more '?'. Now the data just gets *truncated* when it hits a
non-Latin character.

Here is my classpath:

/usr/jdk1.3/jre/lib/rt.jar:/usr/jdk1.3/lib/tools.jar:/usr/jakarta-oro-2.0.1/jakarta-oro-2.0.1.jar:/usr/jakarta-regexp-1.2/jakarta-regexp-1.2.jar:/usr/jakarta-servletapi-3.2/lib/servlet.jar:/usr/jakarta-tomcat-3.2.1/lib/jasper.jar:/usr/jakarta-tomcat-3.2.1/lib/webserver.jar:/usr/Jetspeed-1.1/Jetspeed.jar:/usr/cocoon-1.8/lib/stylebook-1.0-b2.jar:/usr/cocoon-1.8/bin/cocoon.jar:/usr/xerces-1_2_0/xerces.jar:/usr/jakarta-ant/lib/ant.jar:/usr/jakarta-ant/lib/jaxp.jar:/usr/jakarta-ant/lib/optional.jar:/usr/jakarta-ant/lib/parser.jar:/usr/xalan_1_2_D01/xalan.jar:/usr/xalan_1_2_D01/bsf.jar:/usr/xalan_1_2_D01/bsfengines.jar:/usr/aoserv/lib/acme.jar:/usr/aoserv/lib/aocode-public.jar:/usr/aoserv/lib/aoserv-private.jar:/usr/aoserv/lib/damud.jar:/usr/aoserv/lib/danshome.jar:/usr/castor/castor-0.8.8.jar:/usr/cos/lib/cos.jar:/usr/ecs-1.3.3/ecs-1.3.3.jar:/usr/freemarker152/freemarker-jdk1.2.jar:/usr/gnu.regexp-1.0.8/lib/gnu-regexp-1.0.8.jar:/usr/jaf-1.0.1/activation.jar:/usr/jakarta-slide-1.0m4/sli
d
e/lib/webdav.jar:/usr/jakarta-slide-1.0m4/slide/lib/slide.jar:/usr/javachart/jars/kcServlet.jar:/usr/javamail-1.1.3/mail.jar:/usr/jdbc-2.0/jdbc-2.0.jar:/usr/jsse1.0.2/lib/jcert.jar:/usr/jsse1.0.2/lib/jnet.jar:/usr/jsse1.0.2/lib/jsse.jar:/usr/Apache-Jyve-20000907/src/java:/usr/mm.mysql-2.0.4/mm-mysql.jar:/usr/openxml/openxml-1.2.jar:/usr/ozone-0.5.5:/usr/share/pgsql/jdbc7.1-1.2.jar:/usr/pop3-1.1.1/pop3.jar:/usr/soap-2_0/lib/soap.jar:/usr/spfc-0.2.0/bin/spfc.jar:/usr/Apache-Turbine-20000907/src/java:/usr/village-1.3/village-1.3.jar:/usr/webmacro/webmacro.jar:/usr/xang_0_0_2/fesi.zip:/usr/xang_0_0_2/xang.jar:/usr/xmlrpc-java/lib/xmlrpc.jar:/usr/ApacheJSSI-1.1.2/src/java/ApacheJSSI.jar:/usr/interclient/interclient.jar:/usr/PoolMan-1.4.1/lib/PoolMan.jar:/usr/fop-0_13_0/fop_bin_0_13_0.jar:/usr/fop-0_13_0/lib/xml.jar:/wwwgroup/culley/classes:/www/flashcard/classes:/www/banyan/classes

In the same directory as the 7.1-1.2 driver is a copy of the old
driver. Is there someway I could still be hitting the old one?

An interesting anomoly-- When I hit my remote production postgres server
from my local tomcat (normally I'm using tomcat on the same machine as the
database) non-English characters *were* displayed correctly. I was through
the jdbc7.0-1.2.jar driver though! Very confusing...

I'm currently updating my local dinky little linux box with postgres 7.1.3
so my dev environment more closely matches the production. Debugging
should be easier...

culley

At 04:33 PM 9/11/01 -0700, you wrote:

Show quoted text

Culley,

That all looks fine. I don't understand why you would be having problems
with these settings. Are you sure you are using the 7.1 jdbc
drivers? One other thing I can think of to check is to do a 'show
client_encoding'. This should also return UNICODE.

So to restate your problem, you get the 8bit characters returned as ?'s
through jdbc, but it works correctly through psql. Is this a correct
restatement of your problem?

thanks,
--Barry

Culley Harrelson wrote:

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5
All other databases on the server had 0 for encoding.
Unfortunately my dev environment still has postgres 7.0. I updated my
connection pool object and it works on dev but not on prod with 7.1. I
really don't want to update my dev environment untill I can do the whole
machine from scratch but I fear I might have to in order to debug this...
Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#17Barry Lind
barry@xythos.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

If that is the classpath you are using then you should be using the
correct code. Can you sucessfully select the data through psql?

thanks,
--Barry

Culley Harrelson wrote:

No this problem was fixed when I rebuild the database with UNICODE
encoding. No more '?'. Now the data just gets *truncated* when it hits
a non-Latin character.

Here is my classpath:

/usr/jdk1.3/jre/lib/rt.jar:/usr/jdk1.3/lib/tools.jar:/usr/jakarta-oro-2.0.1/jakarta-oro-2.0.1.jar:/usr/jakarta-regexp-1.2/jakarta-regexp-1.2.jar:/usr/jakarta-servletapi-3.2/lib/servlet.jar:/usr/jakarta-tomcat-3.2.1/lib/jasper.jar:/usr/jakarta-tomcat-3.2.1/lib/webserver.jar:/usr/Jetspeed-1.1/Jetspeed.jar:/usr/cocoon-1.8/lib/stylebook-1.0-b2.jar:/usr/cocoon-1.8/bin/cocoon.jar:/usr/xerces-1_2_0/xerces.jar:/usr/jakarta-ant/lib/ant.jar:/usr/jakarta-ant/lib/jaxp.jar:/usr/jakarta-ant/lib/optional.jar:/usr/jakarta-ant/lib/parser.jar:/usr/xalan_1_2_D01/xalan.jar:/usr/xalan_1_2_D01/bsf.jar:/usr/xalan_1_2_D01/bsfengines.jar:/usr/aoserv/lib/acme.jar:/usr/aoserv/lib/aocode-public.jar:/usr/aoserv/lib/aoserv-private.jar:/usr/aoserv/lib/damud.jar:/usr/aoserv/lib/danshome.jar:/usr/castor/castor-0.8.8.jar:/usr/cos/lib/cos.jar:/usr/ecs-1.3.3/ecs-1.3.3.jar:/usr/freemarker152/freemarker-jdk1.2.jar:/usr/gnu.regexp-1.0.8/lib/gnu-regexp-1.0.8.jar:/usr/jaf-1.0.1/activation.jar:/usr/jakarta-slide-1.

0m4/slide/lib/webdav.jar:/usr/jakarta-slide-1.0m4/slide/lib/slide.jar:/usr/javachart/jars/kcServlet.jar:/usr/javamail-1.1.3/mail.jar:*/usr/jdbc-2.0/jdbc-2.0.jar*:/usr/jsse1.0.2/lib/jcert.jar:/usr/jsse1.0.2/lib/jnet.jar:/usr/jsse1.0.2/lib/jsse.jar:/usr/Apache-Jyve-20000907/src/java:/usr/mm.mysql-2.0.4/mm-mysql.jar:/usr/openxml/openxml-1.2.jar:/usr/ozone-0.5.5:*/usr/share/pgsql/jdbc7.1-1.2.jar*:/usr/pop3-1.1.1/pop3.jar:/usr/soap-2_0/lib/soap.jar:/usr/spfc-0.2.0/bin/spfc.jar:/usr/Apache-Turbine-20000907/src/java:/usr/village-1.3/village-1.3.jar:/usr/webmacro/webmacro.jar:/usr/xang_0_0_2/fesi.zip:/usr/xang_0_0_2/xang.jar:/usr/xmlrpc-java/lib/xmlrpc.jar:/usr/ApacheJSSI-1.1.2/src/java/ApacheJSSI.jar:/usr/interclient/interclient.jar:/usr/PoolMan-1.4.1/lib/PoolMan.jar:/usr/fop-0_13_0/fop_bin_0_13_0.jar:/usr/fop-0_13_0/lib/xml.jar:/wwwgroup/culley/classes:/www/flashcard/classes:/www/banyan/classes

Show quoted text

In the same directory as the 7.1-1.2 driver is a copy of the old
driver. Is there someway I could still be hitting the old one?

An interesting anomoly-- When I hit my remote production postgres server
from my local tomcat (normally I'm using tomcat on the same machine as
the database) non-English characters *were* displayed correctly. I was
through the jdbc7.0-1.2.jar driver though! Very confusing...

I'm currently updating my local dinky little linux box with postgres
7.1.3 so my dev environment more closely matches the production.
Debugging should be easier...

culley

At 04:33 PM 9/11/01 -0700, you wrote:

Culley,

That all looks fine. I don't understand why you would be having
problems with these settings. Are you sure you are using the 7.1 jdbc
drivers? One other thing I can think of to check is to do a 'show
client_encoding'. This should also return UNICODE.

So to restate your problem, you get the 8bit characters returned as
?'s through jdbc, but it works correctly through psql. Is this a
correct restatement of your problem?

thanks,
--Barry

Culley Harrelson wrote:

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5
All other databases on the server had 0 for encoding.
Unfortunately my dev environment still has postgres 7.0. I updated
my connection pool object and it works on dev but not on prod with
7.1. I really don't want to update my dev environment untill I can
do the whole machine from scratch but I fear I might have to in order
to debug this...
Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
<http://mail.yahoo.com/&gt;

#18Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#17)
Re: Fwd: Re: unicode in 7.1

Yes, psql works fine and displays non-English characters fine.

Culley

At 06:05 PM 9/11/01 -0700, you wrote:

If that is the classpath you are using then you should be using the
correct code. Can you sucessfully select the data through psql?

thanks,
--Barry

Culley Harrelson wrote:

No this problem was fixed when I rebuild the database with UNICODE
encoding. No more '?'. Now the data just gets *truncated* when it hits
a non-Latin character.
Here is my classpath:
/usr/jdk1.3/jre/lib/rt.jar:/usr/jdk1.3/lib/tools.jar:/usr/jakarta-oro-2.0.1/jakarta-oro-2.0.1.jar:/usr/jakarta-regexp-1.2/jakarta-regexp-1.2.jar:/usr/jakarta-servletapi-3.2/lib/servlet.jar:/usr/jakarta-tomcat-3.2.1/lib/jasper.jar:/usr/jakarta-tomcat-3.2.1/lib/webserver.jar:/usr/Jetspeed-1.1/Jetspeed.jar:/usr/cocoon-1.8/lib/stylebook-1.0-b2.jar:/usr/cocoon-1.8/bin/cocoon.jar:/usr/xerces-1_2_0/xerces.jar:/usr/jakarta-ant/lib/ant.jar:/usr/jakarta-ant/lib/jaxp.jar:/usr/jakarta-ant/lib/optional.jar:/usr/jakarta-ant/lib/parser.jar:/usr/xalan_1_2_D01/xalan.jar:/usr/xalan_1_2_D01/bsf.jar:/usr/xalan_1_2_D01/bsfengines.jar:/usr/aoserv/lib/acme.jar:/usr/aoserv/lib/aocode-public.jar:/usr/aoserv/lib/aoserv-private.jar:/usr/aoserv/lib/damud.jar:/usr/aoserv/lib/danshome.jar:/usr/castor/castor-0.8.8.jar:/usr/cos/lib/cos.jar:/usr/ecs-1.3.3/ecs-1.3.3.jar:/usr/freemarker152/freemarker-jdk1.2.jar:/usr/gnu.regexp-1.0.8/lib/gnu-regexp-1.0.8.jar:/usr/jaf-1.0.1/activation.jar:/usr/jakarta-slide-1.

0m4/slide/lib/webdav.jar:/usr/jakarta-slide-1.0m4/slide/lib/slide.jar:/usr/javachart/jars/kcServlet.jar:/usr/javamail-1.1.3/mail.jar:*/usr/jdbc-2.0/jdbc-2.0.jar*:/usr/jsse1.0.2/lib/jcert.jar:/usr/jsse1.0.2/lib/jnet.jar:/usr/jsse1.0.2/lib/jsse.jar:/usr/Apache-Jyve-20000907/src/java:/usr/mm.mysql-2.0.4/mm-mysql.jar:/usr/openxml/openxml-1.2.jar:/usr/ozone-0.5.5:*/usr/share/pgsql/jdbc7.1-1.2.jar*:/usr/pop3-1.1.1/pop3.jar:/usr/soap-2_0/lib/soap.jar:/usr/spfc-0.2.0/bin/spfc.jar:/usr/Apache-Turbine-20000907/src/java:/usr/village-1.3/village-1.3.jar:/usr/webmacro/webmacro.jar:/usr/xang_0_0_2/fesi.zip:/usr/xang_0_0_2/xang.jar:/usr/xmlrpc-java/lib/xmlrpc.jar:/usr/ApacheJSSI-1.1.2/src/java/ApacheJSSI.jar:/usr/interclient/interclient.jar:/usr/PoolMan-1.4.1/lib/PoolMan.jar:/usr/fop-0_13_0/fop_bin_0_13_0.jar:/usr/fop-0_13_0/lib/xml.jar:/wwwgroup/culley/classes:/www/flashcard/classes:/www/banyan/classes

In the same directory as the 7.1-1.2 driver is a copy of the old
driver. Is there someway I could still be hitting the old one?
An interesting anomoly-- When I hit my remote production postgres server
from my local tomcat (normally I'm using tomcat on the same machine as
the database) non-English characters *were* displayed correctly. I was
through the jdbc7.0-1.2.jar driver though! Very confusing...
I'm currently updating my local dinky little linux box with postgres
7.1.3 so my dev environment more closely matches the production.
Debugging should be easier...
culley

At 04:33 PM 9/11/01 -0700, you wrote:

Culley,

That all looks fine. I don't understand why you would be having
problems with these settings. Are you sure you are using the 7.1 jdbc
drivers? One other thing I can think of to check is to do a 'show
client_encoding'. This should also return UNICODE.

So to restate your problem, you get the 8bit characters returned as ?'s
through jdbc, but it works correctly through psql. Is this a correct
restatement of your problem?

thanks,
--Barry

Culley Harrelson wrote:

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5
All other databases on the server had 0 for encoding.
Unfortunately my dev environment still has postgres 7.0. I updated my
connection pool object and it works on dev but not on prod with 7.1. I
really don't want to update my dev environment untill I can do the
whole machine from scratch but I fear I might have to in order to debug this...
Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
<http://mail.yahoo.com/&gt;

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#19Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

I just finished rebuilding my development environment and have duplicated
the error. I have a database created with the -E UNICODE flag and I am
using the jdbc7.1-1.2.jar driver. Data just gets truncated with a
non-latin character without the charSet property switch, and I get a sql
error when I leave in the property switch. Here is a snap of my code:

--------------------------
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
Properties props = new Properties();
Class.forName ("org.postgresql.Driver");
props.put("user","login");
props.put("password","password");
props.put("charSet","UNICODE");
conn = DriverManager.getConnection("jdbc:postgresql://url", props);

stmt = conn.createStatement();
rs = stmt.executeQuery("select 'TEST' as test");
rs.next();
String dummy = "";
dummy = rs.getString("title");
conn.close();
---------------------------------------

And when I do this I get the following SQL error:

---------------------------------------

Internal Servlet Error:
javax.servlet.ServletException:
at
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:459)
at
_0002fzz_0002ejspzz_jsp_464._jspService(_0002fzz_0002ejspzz_jsp_464.java:97)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.jasper.servlet.JspServlet$JspServletWrapper.service(JspServlet.java:177)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:318)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:391)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.tomcat.core.ServletWrapper.doService(ServletWrapper.java:404)
at org.apache.tomcat.core.Handler.service(Handler.java:286)
at
org.apache.tomcat.core.ServletWrapper.service(ServletWrapper.java:372)
at
org.apache.tomcat.core.ContextManager.internalService(ContextManager.java:797)
at
org.apache.tomcat.core.ContextManager.service(ContextManager.java:743)
at
org.apache.tomcat.service.http.HttpConnectionHandler.processConnection(HttpConnectionHandler.java:210)
at
org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:416)
at
org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:498)
at java.lang.Thread.run(Thread.java:484)
Root cause:
java.sql.SQLException:
at org.postgresql.Connection.ExecSQL(Connection.java:533)
at org.postgresql.jdbc2.Statement.execute(Statement.java:294)
at org.postgresql.jdbc2.Statement.executeQuery(Statement.java:59)
at
_0002fzz_0002ejspzz_jsp_464._jspService(_0002fzz_0002ejspzz_jsp_464.java:77)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:119)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.jasper.servlet.JspServlet$JspServletWrapper.service(JspServlet.java:177)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:318)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:391)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.tomcat.core.ServletWrapper.doService(ServletWrapper.java:404)
at org.apache.tomcat.core.Handler.service(Handler.java:286)
at
org.apache.tomcat.core.ServletWrapper.service(ServletWrapper.java:372)
at
org.apache.tomcat.core.ContextManager.internalService(ContextManager.java:797)
at
org.apache.tomcat.core.ContextManager.service(ContextManager.java:743)
at
org.apache.tomcat.service.http.HttpConnectionHandler.processConnection(HttpConnectionHandler.java:210)
at
org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:416)
at
org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:498)
at java.lang.Thread.run(Thread.java:484)

---------------------------------------

If I take out the encoding line it works but truncates at any non-latin
character.

culley

At 07:33 PM 9/11/01 -0700, you wrote:

Show quoted text

Culley,

I would like to help, but I'm not sure what I can do to help. If there is
anything I can do, please let me know.

thanks,
--Barry

Culley Harrelson wrote:

Yes, psql works fine and displays non-English characters fine.
Culley
At 06:05 PM 9/11/01 -0700, you wrote:

If that is the classpath you are using then you should be using the
correct code. Can you sucessfully select the data through psql?

thanks,
--Barry

Culley Harrelson wrote:

No this problem was fixed when I rebuild the database with UNICODE
encoding. No more '?'. Now the data just gets *truncated* when it
hits a non-Latin character.
Here is my classpath:
/usr/jdk1.3/jre/lib/rt.jar:/usr/jdk1.3/lib/tools.jar:/usr/jakarta-oro-2.0.1/jakarta-oro-2.0.1.jar:/usr/jakarta-regexp-1.2/jakarta-regexp-1.2.jar:/usr/jakarta-servletapi-3.2/lib/servlet.jar:/usr/jakarta-tomcat-3.2.1/lib/jasper.jar:/usr/jakarta-tomcat-3.2.1/lib/webserver.jar:/usr/Jetspeed-1.1/Jetspeed.jar:/usr/cocoon-1.8/lib/stylebook-1.0-b2.jar:/usr/cocoon-1.8/bin/cocoon.jar:/usr/xerces-1_2_0/xerces.jar:/usr/jakarta-ant/lib/ant.jar:/usr/jakarta-ant/lib/jaxp.jar:/usr/jakarta-ant/lib/optional.jar:/usr/jakarta-ant/lib/parser.jar:/usr/xalan_1_2_D01/xalan.jar:/usr/xalan_1_2_D01/bsf.jar:/usr/xalan_1_2_D01/bsfengines.jar:/usr/aoserv/lib/acme.jar:/usr/aoserv/lib/aocode-public.jar:/usr/aoserv/lib/aoserv-private.jar:/usr/aoserv/lib/damud.jar:/usr/aoserv/lib/danshome.jar:/usr/castor/castor-0.8.8.jar:/usr/cos/lib/cos.jar:/usr/ecs-1.3.3/ecs-1.3.3.jar:/usr/freemarker152/freemarker-jdk1.2.jar:/usr/gnu.regexp-1.0.8/lib/gnu-regexp-1.0.8.jar:/usr/jaf-1.0.1/activation.jar:/usr/jakarta-slide-

1.

0m4/slide/lib/webdav.jar:/usr/jakarta-slide-1.0m4/slide/lib/slide.jar:/usr/javachart/jars/kcServlet.jar:/usr/javamail-1.1.3/mail.jar:*/usr/jdbc-2.0/jdbc-2.0.jar*:/usr/jsse1.0.2/lib/jcert.jar:/usr/jsse1.0.2/lib/jnet.jar:/usr/jsse1.0.2/lib/jsse.jar:/usr/Apache-Jyve-20000907/src/java:/usr/mm.mysql-2.0.4/mm-mysql.jar:/usr/openxml/openxml-1.2.jar:/usr/ozone-0.5.5:*/usr/share/pgsql/jdbc7.1-1.2.jar*:/usr/pop3-1.1.1/pop3.jar:/usr/soap-2_0/lib/soap.jar:/usr/spfc-0.2.0/bin/spfc.jar:/usr/Apache-Turbine-20000907/src/java:/usr/village-1.3/village-1.3.jar:/usr/webmacro/webmacro.jar:/usr/xang_0_0_2/fesi.zip:/usr/xang_0_0_2/xang.jar:/usr/xmlrpc-java/lib/xmlrpc.jar:/usr/ApacheJSSI-1.1.2/src/java/ApacheJSSI.jar:/usr/interclient/interclient.jar:/usr/PoolMan-1.4.1/lib/PoolMan.jar:/usr/fop-0_13_0/fop_bin_0_13_0.jar:/usr/fop-0_13_0/lib/xml.jar:/wwwgroup/culley/classes:/www/flashcard/classes:/www/banyan/classes

In the same directory as the 7.1-1.2 driver is a copy of the old
driver. Is there someway I could still be hitting the old one?
An interesting anomoly-- When I hit my remote production postgres
server from my local tomcat (normally I'm using tomcat on the same
machine as the database) non-English characters *were* displayed
correctly. I was through the jdbc7.0-1.2.jar driver though! Very confusing...
I'm currently updating my local dinky little linux box with postgres
7.1.3 so my dev environment more closely matches the production.
Debugging should be easier...
culley

At 04:33 PM 9/11/01 -0700, you wrote:

Culley,

That all looks fine. I don't understand why you would be having
problems with these settings. Are you sure you are using the 7.1 jdbc
drivers? One other thing I can think of to check is to do a 'show
client_encoding'. This should also return UNICODE.

So to restate your problem, you get the 8bit characters returned as
?'s through jdbc, but it works correctly through psql. Is this a
correct restatement of your problem?

thanks,
--Barry

Culley Harrelson wrote:

At 11:45 AM 9/11/01 -0700, you wrote:

select getdatabaseencoding();

UNICODE

select datname, encoding from pg_database;

flashcard | 5
All other databases on the server had 0 for encoding.
Unfortunately my dev environment still has postgres 7.0. I updated
my connection pool object and it works on dev but not on prod with
7.1. I really don't want to update my dev environment untill I can
do the whole machine from scratch but I fear I might have to in order
to debug this...
Culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com
<http://mail.yahoo.com/&gt;

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#20Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Culley Harrelson (#4)
Re: Fwd: Re: unicode in 7.1

Appending to my last post.

I just check the postgres log and It looks like I'm getting a parser error?!?

culley

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

#21Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Culley Harrelson (#11)
#22Barry Lind
barry@xythos.com
In reply to: Culley Harrelson (#4)
#23Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#22)
#24Culley Harrelson
culleyharrelson@yahoo.com
In reply to: Barry Lind (#22)