Bug #943: Server-Encoding from EUC_TW to UTF-8 doesn't work

It turned out that it's a bug with encoding conversion engine of
PostgreSQL. It just failed to find proper entry from a encoding
conversion table because of a integer overflow problem. Since only
maps for EUC_TW have such a huge code point values (for example
0x8eaee7aa), I believe the conversion failure merely occurs with the
particular encoding. Included patches should solve the problem (it is
against PostgreSQL 7.3.2).

BTW, I'm surprised to find the bug since it has been there since 7.2
days.

I'm going to commit the fix to both current and 7.3-stable trees.
--
Tatsuo Ishii

Short Description
Server-Encoding from EUC_TW to UTF-8 doesn't work

Long Description
System: SuSE Linux 8.1, kernel 2.4.19, glibc 2.2.5/glibc-locale 2.2.5
the same error on RedHat 7.3, kernel 2.4.20, glibc2.2.5
postgresql version 7.3.2
description: I loaded Chinese (TW) characters, encoded as UTF-8 into a
database which has UTF-8 encoding with "copy table from 'original'" with psql. Ok.
Than I exit from psql, exported PGCLIENTENCODING=EUC_TW
I started psql, make a "copy table to 'file.EUC_TW'". Ok.
If I convert this file to UTF-8 with iconv -f EUC-TW -t UTF-8 file.EUC_TW file.UTF-8
than file.UTF-8 looks ecaxtly the same as the original.
That means, PostgreSQL converts from UTF-8 to EUC_TW correct.
Now I load the exported file 'file.EUC_TW' back into DB:
"copy table2 from 'file.EUC_TW'", still I did not finish psql,
PGCLIENTENCODING is the same as for "copy to".
Now I get error telling me: "copy: line 1, LocalToUtf: could not convert (0xe5b5) EUC_TW to UTF-8" ... and the characters are missing in table2

Sample Code
UTF-8:
00000000: e795 b6e6 97a5 0ae5 959f e58b 95e4 b8ad
00000010: 2ce4 bd86 e69c 89e9 8caf e8aa a40a

EUC_TW as exported from PostgreSQL and not imported:
00000000: e5b5 c5ca 0ada f6d9 afc4 e32c c8fe c8b4
00000010: f2e3 eba8 0a

*** src/backend/utils/mb/conv.c.orig	2003-04-12 10:03:25.000000000 +0900
--- src/backend/utils/mb/conv.c	2003-04-12 10:16:04.000000000 +0900
***************
*** 313,319 ****

v1 = *(unsigned int *) p1;
v2 = ((pg_utf_to_local *) p2)->utf;
! return (v1 - v2);
}

  /*
--- 313,319 ----

v1 = *(unsigned int *) p1;
v2 = ((pg_utf_to_local *) p2)->utf;
! return (v1 > v2)?1:((v1 == v2)?0:-1);
}

/*
***************
*** 328,334 ****

v1 = *(unsigned int *) p1;
v2 = ((pg_local_to_utf *) p2)->code;
! return (v1 - v2);
}

  /*
--- 328,334 ----

v1 = *(unsigned int *) p1;
v2 = ((pg_local_to_utf *) p2)->code;
! return (v1 > v2)?1:((v1 == v2)?0:-1);
}

/*

I tried also BIG5 encoded data (Trad. Chinese for Taiwan) and got warnings:
WARNING: copy: line 4586, LocalToUtf: could not convert (0xf9d7) BIG5 to UTF-8. Ignored
...
Is this also solved with this fix?

Michael

Tatsuo Ishii wrote:

I tried also BIG5 encoded data (Trad. Chinese for Taiwan) and got warnings:
WARNING: copy: line 4586, LocalToUtf: could not convert (0xf9d7) BIG5 to UTF-8. Ignored
...
Is this also solved with this fix?

No. In your case it seems 0xf9d7 is not a valid BIG5 data, since
there's no corresponding Unicode data for it.
--
Tatsuo Ishii