invalid byte sequence for encoding "UTF8": 0xff

On Mon, Sep 03, 2007 at 01:36:58PM +0530, Ashish Karalkar wrote:

Hello All,

I have a data script which runs fine from PgAdmin SQL Editor,but when I run this from command prompt I get following error:
test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql

psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1: ERROR: invalid byt
e sequence for encoding "UTF8": 0xff
HINT: This error can also happen if the byte sequence does not match the encodi
ng expected by the server, which is controlled by "client_encoding".

Well, the error is correct, that's not a valid UTF-8 character. I seem
to remember someone saying that ooasionally windows puts BOMs in UTF-8
files (which is completely bogus). Check the file using a simple text
editor a check if there are some odd characters at the beginning of the
file.

Have a ncie day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

Ashish Karalkar wrote:

I have a data script which runs fine from PgAdmin SQL
Editor,but when I run this from command prompt I get
following error:

test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql

psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
ERROR: invalid byte sequence for encoding "UTF8": 0xff
HINT: This error can also happen if the byte sequence does
not match the encoding expected by the server, which is
controlled by "client_encoding".

can anybody suggest me what is going wrong.
database encoding :UTF8

PostgreSQL details:

version
--------------------------------------------------------------
PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
(GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

Can you tell us the following:

- What is the client operating system (where you run psql and
PgAdmin III)?
- What is the value of the environment variable PGCLIENTENCODING
set to on the client?
- What does the SQL command "show client_encoding;" return
when you issue it in
a) PgAdmin III
b) psql
- Please create a file that contains only the first line
of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
commands), run the following two (Linux) commands on it:
a) od -t c l
b) od -t x1 l
and show us the output of both commands.

Yours,
Laurenz Albe

----- Original Message -----
From: "Ashish Karalkar" <ashish.karalkar@info-spectrum.com>
To: "Albe Laurenz" <all@adv.magwien.gv.at>
Sent: Monday, September 03, 2007 4:09 PM
Subject: Re: [GENERAL] invalid byte sequence for encoding "UTF8": 0xff

Ashish Karalkar wrote:

I have a data script which runs fine from PgAdmin SQL
Editor,but when I run this from command prompt I get
following error:

test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql

psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
ERROR: invalid byte sequence for encoding "UTF8": 0xff

version
--------------------------------------------------------------
PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
(GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

- What is the client operating system (where you run psql and
PgAdmin III)?

Its Windows XP - PgAdmin III
RHEL 3.4.3-9.EL4-psql (Server Machine)

So I get it that you run "psql" on Windows XP, right?

PGCLIENTENCODING is not set and as per documantation I
think by default it takes value of database i.e. UTF8

- What does the SQL command "show client_encoding;" return
when you issue it in
a) PgAdmin III

UNICODE

b) psql

UTF8

Ok, I suspect that's your problem.
You created QSWEB_100_4_Default_Data.sql by using the "Save" dialog
in PgAdmin III on the Windows machine, right?

Then the file will probably be encoded in Windows-1252.

If your client_encoding is set to UTF8, psql will expect UTF-8
data in the SQL script and complain if it meets wrong ones.

Does the script work as expected when you change the client
encoding to WIN1252?

- Please create a file that contains only the first line
of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
commands), run the following two (Linux) commands on it:
a) od -t c l
b) od -t x1 l
and show us the output of both commands.

[root@localhost qsweb]# od -t c test.sql
0000000 \ s e t O N _ E R R O R _ S

T

0000020 O P
0000022
[root@localhost qsweb]# od -t x1 test.sql
0000000 5c 73 65 74 20 4f 4e 5f 45 52 52 4f 52 5f 53 54
0000020 4f 50
0000022

That's weird, because psql complained about line 1.

Maybe you messed something up by extracting the first line.

Try the following:

- Use binary file transfer and transfer the SQL script to a Linux
machine.

- Run "od -t c -t x1" on the file

- Find the 0xff that psql complains about.

Maybe that helps to locate the problem.
0xff is an unusual Windows-1252 character as well...

Yours,
Laurenz Albe

----- Original Message -----
From: "Albe Laurenz" <all@adv.magwien.gv.at>
To: "Ashish Karalkar *EXTERN*" <ashish.karalkar@info-spectrum.com>
Cc: <pgsql-general@postgresql.org>
Sent: Monday, September 03, 2007 4:54 PM
Subject: RE: [GENERAL] invalid byte sequence for encoding "UTF8": 0xff

Ashish Karalkar wrote:

I have a data script which runs fine from PgAdmin SQL
Editor,but when I run this from command prompt I get
following error:

test=# \i /usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql

psql:/usr/local/pgsql/qsweb1/QSWEB_100_4_Default_Data.sql:1:
ERROR: invalid byte sequence for encoding "UTF8": 0xff

version
--------------------------------------------------------------
PostgreSQL 8.2.0 on i686-pc-linux-gnu, compiled by GCC gcc
(GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

- What is the client operating system (where you run psql and
PgAdmin III)?

Its Windows XP - PgAdmin III
RHEL 3.4.3-9.EL4-psql (Server Machine)

So I get it that you run "psql" on Windows XP, right?

PGCLIENTENCODING is not set and as per documantation I
think by default it takes value of database i.e. UTF8

- What does the SQL command "show client_encoding;" return
when you issue it in
a) PgAdmin III

UNICODE

b) psql

UTF8

Ok, I suspect that's your problem.
You created QSWEB_100_4_Default_Data.sql by using the "Save" dialog
in PgAdmin III on the Windows machine, right?

Then the file will probably be encoded in Windows-1252.

If your client_encoding is set to UTF8, psql will expect UTF-8
data in the SQL script and complain if it meets wrong ones.

Does the script work as expected when you change the client
encoding to WIN1252?

- Please create a file that contains only the first line
of QSWEB_100_4_Default_Data.sql (I call it "l" in the following
commands), run the following two (Linux) commands on it:
a) od -t c l
b) od -t x1 l
and show us the output of both commands.

[root@localhost qsweb]# od -t c test.sql
0000000 \ s e t O N _ E R R O R _ S

T

0000020 O P
0000022
[root@localhost qsweb]# od -t x1 test.sql
0000000 5c 73 65 74 20 4f 4e 5f 45 52 52 4f 52 5f 53 54
0000020 4f 50
0000022

That's weird, because psql complained about line 1.

Maybe you messed something up by extracting the first line.

Try the following:

- Use binary file transfer and transfer the SQL script to a Linux
machine.

- Run "od -t c -t x1" on the file

- Find the 0xff that psql complains about.

Maybe that helps to locate the problem.
0xff is an unusual Windows-1252 character as well...

Hey Thanks Albe it worked.

Yours,
Laurenz Albe