Errors in our encoding conversion tables
There's a discussion over at
/messages/by-id/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz
of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this
and found that indeed, the code will happily translate certain characters
for which there seems to be no justification. I made up a quick script
that would recompute the conversion tables in latin2_and_win1250.c from
the Unicode mapping files in src/backend/utils/mb/Unicode, and what it
computes is shown in the attached diff. (Zeroes in the tables indicate
codes with no translation, for which an error should be thrown.)
Having done that, I thought it would be a good idea to see if we had any
other conversion tables that weren't directly based on the Unicode data.
The only ones I could find were in cyrillic_and_mic.c, and those seem to
be absolutely filled with errors, to the point where I wonder if they were
made from the claimed encodings or some other ones. The attached patch
recomputes those from the Unicode data, too.
None of this data seems to have been touched since Tatsuo-san's original
commit 969e0246, so it looks like we simply didn't vet that submission
closely enough.
I have not attempted to reverify the files in utils/mb/Unicode against the
original Unicode Consortium data, but maybe we ought to do that before
taking any further steps here.
Anyway, what are we going to do about this? I'm concerned that simply
shoving in corrections may cause problems for users. Almost certainly,
we should not back-patch this kind of change.
regards, tom lane
Attachments:
encoding-conversion-corrections.patchtext/x-diff; charset=us-ascii; name=encoding-conversion-corrections.patchDownload+68-66
There's a discussion over at
/messages/by-id/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz
of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this
and found that indeed, the code will happily translate certain characters
for which there seems to be no justification. I made up a quick script
that would recompute the conversion tables in latin2_and_win1250.c from
the Unicode mapping files in src/backend/utils/mb/Unicode, and what it
computes is shown in the attached diff. (Zeroes in the tables indicate
codes with no translation, for which an error should be thrown.)Having done that, I thought it would be a good idea to see if we had any
other conversion tables that weren't directly based on the Unicode data.
The only ones I could find were in cyrillic_and_mic.c, and those seem to
be absolutely filled with errors, to the point where I wonder if they were
made from the claimed encodings or some other ones. The attached patch
recomputes those from the Unicode data, too.None of this data seems to have been touched since Tatsuo-san's original
commit 969e0246, so it looks like we simply didn't vet that submission
closely enough.I have not attempted to reverify the files in utils/mb/Unicode against the
original Unicode Consortium data, but maybe we ought to do that before
taking any further steps here.Anyway, what are we going to do about this? I'm concerned that simply
shoving in corrections may cause problems for users. Almost certainly,
we should not back-patch this kind of change.
I have started to looking into it. I wonder how do you create the part
of your patch:
*** 154,163 ****
win12502mic(const unsigned char *l, unsigned char *p, int len)
{
static const unsigned char win1250_2_iso88592[] = {
! 0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
! 0x88, 0x89, 0xA9, 0x8B, 0xA6, 0xAB, 0xAE, 0xAC,
! 0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97,
! 0x98, 0x99, 0xB9, 0x9B, 0xB6, 0xBB, 0xBE, 0xBC,
0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7,
0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF,
0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00,
--- 154,163 ----
win12502mic(const unsigned char *l, unsigned char *p, int len)
{
static const unsigned char win1250_2_iso88592[] = {
! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
! 0x00, 0x00, 0xA9, 0x00, 0xA6, 0xAB, 0xAE, 0xAC,
! 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
! 0x00, 0x00, 0xB9, 0x00, 0xB6, 0xBB, 0xBE, 0xBC,
0xA0, 0xB7, 0xA2, 0xA3, 0xA4, 0xA1, 0x00, 0xA7,
0xA8, 0x00, 0xAA, 0x00, 0x00, 0xAD, 0x00, 0xAF,
0xB0, 0x00, 0xB2, 0xB3, 0xB4, 0x00, 0x00, 0x00,In the above you seem to disable the conversion from 0x96 of win1250
to ISO-8859-2 by using the Unicode mapping files in
src/backend/utils/mb/Unicode. But the corresponding mapping file
(iso8859_2_to_utf8.amp) does include following entry:
{0x0096, 0xc296},
How do you know 0x96 should be removed from the conversion?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Tatsuo Ishii <ishii@postgresql.org> writes:
I have started to looking into it. I wonder how do you create the part
of your patch:
The code I used is below.
In the above you seem to disable the conversion from 0x96 of win1250
to ISO-8859-2 by using the Unicode mapping files in
src/backend/utils/mb/Unicode. But the corresponding mapping file
(iso8859_2_to_utf8.amp) does include following entry:
{0x0096, 0xc296},
How do you know 0x96 should be removed from the conversion?
Right, but there is no mapping in the win1250-utf8 files that matches
U+C296. The complaint over in the other thread is precisely that we
have no business translating 0x96 in WIN1250 to this character. What
WIN1250 0x96 could translate to is U+E28093 (at least, according to
win1250_to_utf8.map) but that Unicode character has no equivalent in
LATIN2.
AFAICS, whoever made these tables just arbitrarily decided that 0x96
in WIN1250 could be mapped to 0x96 in LATIN2, and likewise for a number
of other codes; but those are false equivalences, as you find out if
you try to perform the same conversion via other encoding conversion
paths, ie convert to UTF8 and then to the other encoding.
regards, tom lane
Attachments:
buildmaps.ctext/x-c; charset=us-ascii; name=buildmaps.cDownload
Tom Lane wrote:
There's a discussion over at
/messages/by-id/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz
of an apparent error in our WIN1250 -> LATIN2 conversion. I looked into this
and found that indeed, the code will happily translate certain characters
for which there seems to be no justification. I made up a quick script
that would recompute the conversion tables in latin2_and_win1250.c from
the Unicode mapping files in src/backend/utils/mb/Unicode, and what it
computes is shown in the attached diff. (Zeroes in the tables indicate
codes with no translation, for which an error should be thrown.)Having done that, I thought it would be a good idea to see if we had any
other conversion tables that weren't directly based on the Unicode data.
The only ones I could find were in cyrillic_and_mic.c, and those seem to
be absolutely filled with errors, to the point where I wonder if they were
made from the claimed encodings or some other ones. The attached patch
recomputes those from the Unicode data, too.None of this data seems to have been touched since Tatsuo-san's original
commit 969e0246, so it looks like we simply didn't vet that submission
closely enough.I have not attempted to reverify the files in utils/mb/Unicode against the
original Unicode Consortium data, but maybe we ought to do that before
taking any further steps here.Anyway, what are we going to do about this? I'm concerned that simply
shoving in corrections may cause problems for users. Almost certainly,
we should not back-patch this kind of change.
Thanks for picking this up.
I agree with your proposed fix, the only thing that makes me feel uncomfortable
is that you get error messages like:
ERROR: character with byte sequence 0x96 in encoding "WIN1250" has no equivalent in encoding "MULE_INTERNAL"
which is a bit misleading.
But the main thing is that no corrupt data can be entered.
I can understand the reluctance to back-patch; nobody likes his
application to suddenly fail after a minor database upgrade.
However, the people who would fail if this were back-patched are
people who will certainly run into trouble if they
a) upgrade to a release where this is fixed or
b) try to convert their database to, say, UTF8.
The least thing we should do is stick a fat warning into the release notes
of the first version where this is fixed, along with some guidelines what
to do (though I am afraid that there is not much more helpful to say than
"If your database encoding is X and data have been entered with client_encoding Y,
fix your data in the old system").
But I think that this fix should be applied to 9.6.
PostgreSQL has a strong reputation for being strict about correct encoding
(not saying that everybody appreciates that), and I think we shouldn't mar
that reputation.
Yours,
Laurenz Albe
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Albe Laurenz <laurenz.albe@wien.gv.at> writes:
I agree with your proposed fix, the only thing that makes me feel uncomfortable
is that you get error messages like:
ERROR: character with byte sequence 0x96 in encoding "WIN1250" has no equivalent in encoding "MULE_INTERNAL"
Hm, yeah. It's pretty silly that this code uses a double conversion when
it has a table that would work fine for a direct conversion, anyway. But
I think improving that is a separate question, especially since the above
behavior occurred already for a few code points.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
I wrote:
I have not attempted to reverify the files in utils/mb/Unicode against the
original Unicode Consortium data, but maybe we ought to do that before
taking any further steps here.
I downloaded the mapping files from unicode.org and attempted to verify
that the Unicode/*.map files could be reproduced from the stated sources.
Most of them are okay, but I failed to verify these:
euc_cn_to_utf8.map utf8_to_euc_cn.map
Could not find the reference file GB2312.TXT; it is not at unicode.org
gb18030_to_utf8.map utf8_to_gb18030.map
Could not find the reference file gb-18030-2000.xml, whose origin is
unstated anyway.
euc_jp_to_utf8.map utf8_to_euc_jp.map
euc_kr_to_utf8.map utf8_to_euc_kr.map
johab_to_utf8.map utf8_to_johab.map
uhc_to_utf8.map utf8_to_uhc.map
These four all have minor to significant differences from what I got by
running the generation scripts. See attached diffs.
utf8_to_sjis.map
It's very disturbing that this fails to verify when its allegedly inverse
file does verify; either the script is broken or somebody did sloppy
manual editing.
Anyway, this seems to mean that it's okay to go ahead with fixing the
encoding conversion discrepancies I complained of yesterday; the data
those proposed diffs are based on is solid. But we've evidently got
a number of other issues with these Far Eastern encodings.
regards, tom lane
*** euc_jp_to_utf8.map.orig Fri May 15 17:56:06 2015
--- euc_jp_to_utf8.map Fri Nov 27 16:25:37 2015
***************
*** 1,6 ****
/* src/backend/utils/mb/Unicode/euc_jp_to_utf8.map */! static const pg_local_to_utf LUmapEUC_JP[] = {
{0x8ea1, 0xefbda1},
{0x8ea2, 0xefbda2},
{0x8ea3, 0xefbda3},
--- 1,6 ----
/* src/backend/utils/mb/Unicode/euc_jp_to_utf8.map */! static const pg_local_to_utf LUmapEUC_JP[ 13007 ] = {
{0x8ea1, 0xefbda1},
{0x8ea2, 0xefbda2},
{0x8ea3, 0xefbda3},
***************
*** 95,103 ****
{0xa1bd, 0xe28095},
{0xa1be, 0xe28090},
{0xa1bf, 0xefbc8f},
! {0xa1c0, 0xefbcbc},
! {0xa1c1, 0xefbd9e},
! {0xa1c2, 0xe288a5},
{0xa1c3, 0xefbd9c},
{0xa1c4, 0xe280a6},
{0xa1c5, 0xe280a5},
--- 95,102 ----
{0xa1bd, 0xe28095},
{0xa1be, 0xe28090},
{0xa1bf, 0xefbc8f},
! {0xa1c1, 0xe3809c},
! {0xa1c2, 0xe28096},
{0xa1c3, 0xefbd9c},
{0xa1c4, 0xe280a6},
{0xa1c5, 0xe280a5},
***************
*** 124,130 ****
{0xa1da, 0xe38090},
{0xa1db, 0xe38091},
{0xa1dc, 0xefbc8b},
! {0xa1dd, 0xefbc8d},
{0xa1de, 0xc2b1},
{0xa1df, 0xc397},
{0xa1e0, 0xc3b7},
--- 123,129 ----
{0xa1da, 0xe38090},
{0xa1db, 0xe38091},
{0xa1dc, 0xefbc8b},
! {0xa1dd, 0xe28892},
{0xa1de, 0xc2b1},
{0xa1df, 0xc397},
{0xa1e0, 0xc3b7},
***************
*** 144,151 ****
{0xa1ee, 0xe28483},
{0xa1ef, 0xefbfa5},
{0xa1f0, 0xefbc84},
! {0xa1f1, 0xefbfa0},
! {0xa1f2, 0xefbfa1},
{0xa1f3, 0xefbc85},
{0xa1f4, 0xefbc83},
{0xa1f5, 0xefbc86},
--- 143,150 ----
{0xa1ee, 0xe28483},
{0xa1ef, 0xefbfa5},
{0xa1f0, 0xefbc84},
! {0xa1f1, 0xc2a2},
! {0xa1f2, 0xc2a3},
{0xa1f3, 0xefbc85},
{0xa1f4, 0xefbc83},
{0xa1f5, 0xefbc86},
***************
*** 182,188 ****
{0xa2c1, 0xe288a9},
{0xa2ca, 0xe288a7},
{0xa2cb, 0xe288a8},
! {0xa2cc, 0xefbfa2},
{0xa2cd, 0xe28792},
{0xa2ce, 0xe28794},
{0xa2cf, 0xe28880},
--- 181,187 ----
{0xa2c1, 0xe288a9},
{0xa2ca, 0xe288a7},
{0xa2cb, 0xe288a8},
! {0xa2cc, 0xc2ac},
{0xa2cd, 0xe28792},
{0xa2ce, 0xe28794},
{0xa2cf, 0xe28880},
***************
*** 588,676 ****
{0xa8be, 0xe294a5},
{0xa8bf, 0xe294b8},
{0xa8c0, 0xe29582},
- {0xada1, 0xe291a0},
- {0xada2, 0xe291a1},
- {0xada3, 0xe291a2},
- {0xada4, 0xe291a3},
- {0xada5, 0xe291a4},
- {0xada6, 0xe291a5},
- {0xada7, 0xe291a6},
- {0xada8, 0xe291a7},
- {0xada9, 0xe291a8},
- {0xadaa, 0xe291a9},
- {0xadab, 0xe291aa},
- {0xadac, 0xe291ab},
- {0xadad, 0xe291ac},
- {0xadae, 0xe291ad},
- {0xadaf, 0xe291ae},
- {0xadb0, 0xe291af},
- {0xadb1, 0xe291b0},
- {0xadb2, 0xe291b1},
- {0xadb3, 0xe291b2},
- {0xadb4, 0xe291b3},
- {0xadb5, 0xe285a0},
- {0xadb6, 0xe285a1},
- {0xadb7, 0xe285a2},
- {0xadb8, 0xe285a3},
- {0xadb9, 0xe285a4},
- {0xadba, 0xe285a5},
- {0xadbb, 0xe285a6},
- {0xadbc, 0xe285a7},
- {0xadbd, 0xe285a8},
- {0xadbe, 0xe285a9},
- {0xadc0, 0xe38d89},
- {0xadc1, 0xe38c94},
- {0xadc2, 0xe38ca2},
- {0xadc3, 0xe38d8d},
- {0xadc4, 0xe38c98},
- {0xadc5, 0xe38ca7},
- {0xadc6, 0xe38c83},
- {0xadc7, 0xe38cb6},
- {0xadc8, 0xe38d91},
- {0xadc9, 0xe38d97},
- {0xadca, 0xe38c8d},
- {0xadcb, 0xe38ca6},
- {0xadcc, 0xe38ca3},
- {0xadcd, 0xe38cab},
- {0xadce, 0xe38d8a},
- {0xadcf, 0xe38cbb},
- {0xadd0, 0xe38e9c},
- {0xadd1, 0xe38e9d},
- {0xadd2, 0xe38e9e},
- {0xadd3, 0xe38e8e},
- {0xadd4, 0xe38e8f},
- {0xadd5, 0xe38f84},
- {0xadd6, 0xe38ea1},
- {0xaddf, 0xe38dbb},
- {0xade0, 0xe3809d},
- {0xade1, 0xe3809f},
- {0xade2, 0xe28496},
- {0xade3, 0xe38f8d},
- {0xade4, 0xe284a1},
- {0xade5, 0xe38aa4},
- {0xade6, 0xe38aa5},
- {0xade7, 0xe38aa6},
- {0xade8, 0xe38aa7},
- {0xade9, 0xe38aa8},
- {0xadea, 0xe388b1},
- {0xadeb, 0xe388b2},
- {0xadec, 0xe388b9},
- {0xaded, 0xe38dbe},
- {0xadee, 0xe38dbd},
- {0xadef, 0xe38dbc},
- {0xadf0, 0xe28992},
- {0xadf1, 0xe289a1},
- {0xadf2, 0xe288ab},
- {0xadf3, 0xe288ae},
- {0xadf4, 0xe28891},
- {0xadf5, 0xe2889a},
- {0xadf6, 0xe28aa5},
- {0xadf7, 0xe288a0},
- {0xadf8, 0xe2889f},
- {0xadf9, 0xe28abf},
- {0xadfa, 0xe288b5},
- {0xadfb, 0xe288a9},
- {0xadfc, 0xe288aa},
{0xb0a1, 0xe4ba9c},
{0xb0a2, 0xe59496},
{0xb0a3, 0xe5a883},
--- 587,592 ----
***************
*** 7037,7043 ****
{0x8fa2b8, 0xce84},
{0x8fa2b9, 0xce85},
{0x8fa2c2, 0xc2a1},
! {0x8fa2c3, 0xefbfa4},
{0x8fa2c4, 0xc2bf},
{0x8fa2eb, 0xc2ba},
{0x8fa2ec, 0xc2aa},
--- 6953,6959 ----
{0x8fa2b8, 0xce84},
{0x8fa2b9, 0xce85},
{0x8fa2c2, 0xc2a1},
! {0x8fa2c3, 0xc2a6},
{0x8fa2c4, 0xc2bf},
{0x8fa2eb, 0xc2ba},
{0x8fa2ec, 0xc2aa},
***************
*** 13091,13201 ****
{0x8fede0, 0xe9bea1},
{0x8fede1, 0xe9bea2},
{0x8fede2, 0xe9bea3},
! {0x8fede3, 0xe9bea5},
! {0x8ff3f3, 0xe285b0},
! {0x8ff3f4, 0xe285b1},
! {0x8ff3f5, 0xe285b2},
! {0x8ff3f6, 0xe285b3},
! {0x8ff3f7, 0xe285b4},
! {0x8ff3f8, 0xe285b5},
! {0x8ff3f9, 0xe285b6},
! {0x8ff3fa, 0xe285b7},
! {0x8ff3fb, 0xe285b8},
! {0x8ff3fc, 0xe285b9},
! {0x8ff3fd, 0xe285a0},
! {0x8ff3fe, 0xe285a1},
! {0x8ff4a1, 0xe285a2},
! {0x8ff4a2, 0xe285a3},
! {0x8ff4a3, 0xe285a4},
! {0x8ff4a4, 0xe285a5},
! {0x8ff4a5, 0xe285a6},
! {0x8ff4a6, 0xe285a7},
! {0x8ff4a7, 0xe285a8},
! {0x8ff4a8, 0xe285a9},
! {0x8ff4a9, 0xefbc87},
! {0x8ff4aa, 0xefbc82},
! {0x8ff4ab, 0xe388b1},
! {0x8ff4ac, 0xe28496},
! {0x8ff4ad, 0xe284a1},
! {0x8ff4ae, 0xe782bb},
! {0x8ff4af, 0xe4bbbc},
! {0x8ff4b0, 0xe583b4},
! {0x8ff4b1, 0xe587ac},
! {0x8ff4b2, 0xe58c87},
! {0x8ff4b3, 0xe58ca4},
! {0x8ff4b4, 0xefa88e},
! {0x8ff4b5, 0xe5928a},
! {0x8ff4b6, 0xe59d99},
! {0x8ff4b7, 0xefa88f},
! {0x8ff4b8, 0xefa890},
! {0x8ff4b9, 0xe5a29e},
! {0x8ff4ba, 0xe5afac},
! {0x8ff4bb, 0xe5b3b5},
! {0x8ff4bc, 0xe5b593},
! {0x8ff4bd, 0xefa891},
! {0x8ff4be, 0xe5beb7},
! {0x8ff4bf, 0xe68285},
! {0x8ff4c0, 0xe684a0},
! {0x8ff4c1, 0xe6958e},
! {0x8ff4c2, 0xe698bb},
! {0x8ff4c3, 0xe699a5},
! {0x8ff4c4, 0xefa892},
! {0x8ff4c5, 0xefa4a9},
! {0x8ff4c6, 0xe6a081},
! {0x8ff4c7, 0xefa893},
! {0x8ff4c8, 0xefa894},
! {0x8ff4c9, 0xe6a9ab},
! {0x8ff4ca, 0xe6aba2},
! {0x8ff4cb, 0xe6b7b8},
! {0x8ff4cc, 0xe6b7b2},
! {0x8ff4cd, 0xe780a8},
! {0x8ff4ce, 0xefa895},
! {0x8ff4cf, 0xefa896},
! {0x8ff4d0, 0xe79481},
! {0x8ff4d1, 0xe79a82},
! {0x8ff4d2, 0xe79a9e},
! {0x8ff4d3, 0xefa897},
! {0x8ff4d4, 0xe7a4b0},
! {0x8ff4d5, 0xefa898},
! {0x8ff4d6, 0xefa899},
! {0x8ff4d7, 0xefa89a},
! {0x8ff4d8, 0xefa89b},
! {0x8ff4d9, 0xe7aba7},
! {0x8ff4da, 0xefa89c},
! {0x8ff4db, 0xefa89d},
! {0x8ff4dc, 0xe7b6a0},
! {0x8ff4dd, 0xe7b796},
! {0x8ff4de, 0xefa89e},
! {0x8ff4df, 0xe88da2},
! {0x8ff4e0, 0xefa89f},
! {0x8ff4e1, 0xe896b0},
! {0x8ff4e2, 0xefa8a0},
! {0x8ff4e3, 0xefa8a1},
! {0x8ff4e4, 0xe8a087},
! {0x8ff4e5, 0xefa8a2},
! {0x8ff4e6, 0xe8adbf},
! {0x8ff4e7, 0xe8b3b4},
! {0x8ff4e8, 0xe8b5b6},
! {0x8ff4e9, 0xefa8a3},
! {0x8ff4ea, 0xefa8a4},
! {0x8ff4eb, 0xefa8a5},
! {0x8ff4ec, 0xe9839e},
! {0x8ff4ed, 0xefa8a6},
! {0x8ff4ee, 0xe98495},
! {0x8ff4ef, 0xefa8a7},
! {0x8ff4f0, 0xefa8a8},
! {0x8ff4f1, 0xe99692},
! {0x8ff4f2, 0xefa79c},
! {0x8ff4f3, 0xefa8a9},
! {0x8ff4f4, 0xe99cbb},
! {0x8ff4f5, 0xe99d8d},
! {0x8ff4f6, 0xe99d91},
! {0x8ff4f7, 0xefa8aa},
! {0x8ff4f8, 0xefa8ab},
! {0x8ff4f9, 0xefa8ac},
! {0x8ff4fa, 0xe9a69e},
! {0x8ff4fb, 0xe9ab99},
! {0x8ff4fc, 0xe9adb2},
! {0x8ff4fd, 0xefa8ad},
! {0x8ff4fe, 0xe9bb91},
};
--- 13007,13011 ----
{0x8fede0, 0xe9bea1},
{0x8fede1, 0xe9bea2},
{0x8fede2, 0xe9bea3},
! {0x8fede3, 0xe9bea5}
};
*** euc_kr_to_utf8.map.orig Fri May 15 17:56:06 2015
--- euc_kr_to_utf8.map Fri Nov 27 16:25:41 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapEUC_KR[ 8227 ] = {
{0xa1a1, 0xe38080},
{0xa1a2, 0xe38081},
{0xa1a3, 0xe38082},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/euc_kr_to_utf8.map */
!
! static const pg_local_to_utf LUmapEUC_KR[ 8224 ] = {
{0xa1a1, 0xe38080},
{0xa1a2, 0xe38081},
{0xa1a3, 0xe38082},
***************
*** 162,170 ****
{0xa2e3, 0xe38f82},
{0xa2e4, 0xe38f98},
{0xa2e5, 0xe284a1},
- {0xa2e6, 0xe282ac},
- {0xa2e7, 0xc2ae},
- {0xa2e8, 0xe389be},
{0xa3a1, 0xefbc81},
{0xa3a2, 0xefbc82},
{0xa3a3, 0xefbc83},
--- 164,169 ----
*** johab_to_utf8.map.orig Fri May 15 17:56:06 2015
--- johab_to_utf8.map Fri Nov 27 16:26:10 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapJOHAB[ 17049 ] = {
{0x8444, 0xe384b3},
{0x8446, 0xe384b5},
{0x8447, 0xe384b6},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/johab_to_utf8.map */
!
! static const pg_local_to_utf LUmapJOHAB[ 17046 ] = {
{0x8444, 0xe384b3},
{0x8446, 0xe384b5},
{0x8447, 0xe384b6},
***************
*** 11385,11393 ****
{0xd9e3, 0xe38f82},
{0xd9e4, 0xe38f98},
{0xd9e5, 0xe284a1},
- {0xd9e6, 0xe282ac},
- {0xd9e7, 0xc2ae},
- {0xd9e8, 0xe389be},
{0xda31, 0xefbc81},
{0xda32, 0xefbc82},
{0xda33, 0xefbc83},
--- 11387,11392 ----
*** uhc_to_utf8.map.orig Fri May 15 17:56:06 2015
--- uhc_to_utf8.map Fri Nov 27 16:26:11 2015
***************
*** 1,4 ****
! static const pg_local_to_utf LUmapUHC[ 17237 ] = {
{0x8141, 0xeab082},
{0x8142, 0xeab083},
{0x8143, 0xeab085},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/uhc_to_utf8.map */
!
! static const pg_local_to_utf LUmapUHC[ 17048 ] = {
{0x8141, 0xeab082},
{0x8142, 0xeab083},
{0x8143, 0xeab085},
***************
*** 6028,6034 ****
{0xa2e5, 0xe284a1},
{0xa2e6, 0xe282ac},
{0xa2e7, 0xc2ae},
- {0xa2e8, 0xe389be},
{0xa341, 0xeca5b1},
{0xa342, 0xeca5b2},
{0xa343, 0xeca5b3},
--- 6030,6035 ----
***************
*** 12160,12259 ****
{0xc8fc, 0xed9e99},
{0xc8fd, 0xed9e9b},
{0xc8fe, 0xed9e9d},
- {0xc9a1, 0xee8080},
- {0xc9a2, 0xee8081},
- {0xc9a3, 0xee8082},
- {0xc9a4, 0xee8083},
- {0xc9a5, 0xee8084},
- {0xc9a6, 0xee8085},
- {0xc9a7, 0xee8086},
- {0xc9a8, 0xee8087},
- {0xc9a9, 0xee8088},
- {0xc9aa, 0xee8089},
- {0xc9ab, 0xee808a},
- {0xc9ac, 0xee808b},
- {0xc9ad, 0xee808c},
- {0xc9ae, 0xee808d},
- {0xc9af, 0xee808e},
- {0xc9b0, 0xee808f},
- {0xc9b1, 0xee8090},
- {0xc9b2, 0xee8091},
- {0xc9b3, 0xee8092},
- {0xc9b4, 0xee8093},
- {0xc9b5, 0xee8094},
- {0xc9b6, 0xee8095},
- {0xc9b7, 0xee8096},
- {0xc9b8, 0xee8097},
- {0xc9b9, 0xee8098},
- {0xc9ba, 0xee8099},
- {0xc9bb, 0xee809a},
- {0xc9bc, 0xee809b},
- {0xc9bd, 0xee809c},
- {0xc9be, 0xee809d},
- {0xc9bf, 0xee809e},
- {0xc9c0, 0xee809f},
- {0xc9c1, 0xee80a0},
- {0xc9c2, 0xee80a1},
- {0xc9c3, 0xee80a2},
- {0xc9c4, 0xee80a3},
- {0xc9c5, 0xee80a4},
- {0xc9c6, 0xee80a5},
- {0xc9c7, 0xee80a6},
- {0xc9c8, 0xee80a7},
- {0xc9c9, 0xee80a8},
- {0xc9ca, 0xee80a9},
- {0xc9cb, 0xee80aa},
- {0xc9cc, 0xee80ab},
- {0xc9cd, 0xee80ac},
- {0xc9ce, 0xee80ad},
- {0xc9cf, 0xee80ae},
- {0xc9d0, 0xee80af},
- {0xc9d1, 0xee80b0},
- {0xc9d2, 0xee80b1},
- {0xc9d3, 0xee80b2},
- {0xc9d4, 0xee80b3},
- {0xc9d5, 0xee80b4},
- {0xc9d6, 0xee80b5},
- {0xc9d7, 0xee80b6},
- {0xc9d8, 0xee80b7},
- {0xc9d9, 0xee80b8},
- {0xc9da, 0xee80b9},
- {0xc9db, 0xee80ba},
- {0xc9dc, 0xee80bb},
- {0xc9dd, 0xee80bc},
- {0xc9de, 0xee80bd},
- {0xc9df, 0xee80be},
- {0xc9e0, 0xee80bf},
- {0xc9e1, 0xee8180},
- {0xc9e2, 0xee8181},
- {0xc9e3, 0xee8182},
- {0xc9e4, 0xee8183},
- {0xc9e5, 0xee8184},
- {0xc9e6, 0xee8185},
- {0xc9e7, 0xee8186},
- {0xc9e8, 0xee8187},
- {0xc9e9, 0xee8188},
- {0xc9ea, 0xee8189},
- {0xc9eb, 0xee818a},
- {0xc9ec, 0xee818b},
- {0xc9ed, 0xee818c},
- {0xc9ee, 0xee818d},
- {0xc9ef, 0xee818e},
- {0xc9f0, 0xee818f},
- {0xc9f1, 0xee8190},
- {0xc9f2, 0xee8191},
- {0xc9f3, 0xee8192},
- {0xc9f4, 0xee8193},
- {0xc9f5, 0xee8194},
- {0xc9f6, 0xee8195},
- {0xc9f7, 0xee8196},
- {0xc9f8, 0xee8197},
- {0xc9f9, 0xee8198},
- {0xc9fa, 0xee8199},
- {0xc9fb, 0xee819a},
- {0xc9fc, 0xee819b},
- {0xc9fd, 0xee819c},
- {0xc9fe, 0xee819d},
{0xcaa1, 0xe4bcbd},
{0xcaa2, 0xe4bdb3},
{0xcaa3, 0xe58187},
--- 12161,12166 ----
***************
*** 17141,17239 ****
{0xfdfb, 0xe7a6a7},
{0xfdfc, 0xe7a880},
{0xfdfd, 0xe7beb2},
! {0xfdfe, 0xe8a9b0},
! {0xfea1, 0xee819e},
! {0xfea2, 0xee819f},
! {0xfea3, 0xee81a0},
! {0xfea4, 0xee81a1},
! {0xfea5, 0xee81a2},
! {0xfea6, 0xee81a3},
! {0xfea7, 0xee81a4},
! {0xfea8, 0xee81a5},
! {0xfea9, 0xee81a6},
! {0xfeaa, 0xee81a7},
! {0xfeab, 0xee81a8},
! {0xfeac, 0xee81a9},
! {0xfead, 0xee81aa},
! {0xfeae, 0xee81ab},
! {0xfeaf, 0xee81ac},
! {0xfeb0, 0xee81ad},
! {0xfeb1, 0xee81ae},
! {0xfeb2, 0xee81af},
! {0xfeb3, 0xee81b0},
! {0xfeb4, 0xee81b1},
! {0xfeb5, 0xee81b2},
! {0xfeb6, 0xee81b3},
! {0xfeb7, 0xee81b4},
! {0xfeb8, 0xee81b5},
! {0xfeb9, 0xee81b6},
! {0xfeba, 0xee81b7},
! {0xfebb, 0xee81b8},
! {0xfebc, 0xee81b9},
! {0xfebd, 0xee81ba},
! {0xfebe, 0xee81bb},
! {0xfebf, 0xee81bc},
! {0xfec0, 0xee81bd},
! {0xfec1, 0xee81be},
! {0xfec2, 0xee81bf},
! {0xfec3, 0xee8280},
! {0xfec4, 0xee8281},
! {0xfec5, 0xee8282},
! {0xfec6, 0xee8283},
! {0xfec7, 0xee8284},
! {0xfec8, 0xee8285},
! {0xfec9, 0xee8286},
! {0xfeca, 0xee8287},
! {0xfecb, 0xee8288},
! {0xfecc, 0xee8289},
! {0xfecd, 0xee828a},
! {0xfece, 0xee828b},
! {0xfecf, 0xee828c},
! {0xfed0, 0xee828d},
! {0xfed1, 0xee828e},
! {0xfed2, 0xee828f},
! {0xfed3, 0xee8290},
! {0xfed4, 0xee8291},
! {0xfed5, 0xee8292},
! {0xfed6, 0xee8293},
! {0xfed7, 0xee8294},
! {0xfed8, 0xee8295},
! {0xfed9, 0xee8296},
! {0xfeda, 0xee8297},
! {0xfedb, 0xee8298},
! {0xfedc, 0xee8299},
! {0xfedd, 0xee829a},
! {0xfede, 0xee829b},
! {0xfedf, 0xee829c},
! {0xfee0, 0xee829d},
! {0xfee1, 0xee829e},
! {0xfee2, 0xee829f},
! {0xfee3, 0xee82a0},
! {0xfee4, 0xee82a1},
! {0xfee5, 0xee82a2},
! {0xfee6, 0xee82a3},
! {0xfee7, 0xee82a4},
! {0xfee8, 0xee82a5},
! {0xfee9, 0xee82a6},
! {0xfeea, 0xee82a7},
! {0xfeeb, 0xee82a8},
! {0xfeec, 0xee82a9},
! {0xfeed, 0xee82aa},
! {0xfeee, 0xee82ab},
! {0xfeef, 0xee82ac},
! {0xfef0, 0xee82ad},
! {0xfef1, 0xee82ae},
! {0xfef2, 0xee82af},
! {0xfef3, 0xee82b0},
! {0xfef4, 0xee82b1},
! {0xfef5, 0xee82b2},
! {0xfef6, 0xee82b3},
! {0xfef7, 0xee82b4},
! {0xfef8, 0xee82b5},
! {0xfef9, 0xee82b6},
! {0xfefa, 0xee82b7},
! {0xfefb, 0xee82b8},
! {0xfefc, 0xee82b9},
! {0xfefd, 0xee82ba},
! {0xfefe, 0xee82bb}
};
--- 17048,17052 ----
{0xfdfb, 0xe7a6a7},
{0xfdfc, 0xe7a880},
{0xfdfd, 0xe7beb2},
! {0xfdfe, 0xe8a9b0}
};
*** utf8_to_euc_jp.map.orig Fri May 15 17:56:06 2015
--- utf8_to_euc_jp.map Fri Nov 27 16:25:37 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapEUC_JP[ 13175 ] = {
{0xc2a1, 0x8fa2c2},
{0xc2a4, 0x8fa2f0},
{0xc2a6, 0x8fa2c3},
{0xc2a7, 0xa1f8},
{0xc2a8, 0xa1af},
{0xc2a9, 0x8fa2ed},
{0xc2aa, 0x8fa2ec},
{0xc2ae, 0x8fa2ee},
{0xc2af, 0x8fa2b4},
{0xc2b0, 0xa1eb},
--- 1,16 ----
! /* src/backend/utils/mb/Unicode/utf8_to_euc_jp.map */
!
! static const pg_utf_to_local ULmapEUC_JP[ 13007 ] = {
{0xc2a1, 0x8fa2c2},
+ {0xc2a2, 0xa1f1},
+ {0xc2a3, 0xa1f2},
{0xc2a4, 0x8fa2f0},
{0xc2a6, 0x8fa2c3},
{0xc2a7, 0xa1f8},
{0xc2a8, 0xa1af},
{0xc2a9, 0x8fa2ed},
{0xc2aa, 0x8fa2ec},
+ {0xc2ac, 0xa2cc},
{0xc2ae, 0x8fa2ee},
{0xc2af, 0x8fa2b4},
{0xc2b0, 0xa1eb},
***************
*** 386,391 ****
--- 391,397 ----
{0xd19f, 0x8fa7fe},
{0xe28090, 0xa1be},
{0xe28095, 0xa1bd},
+ {0xe28096, 0xa1c2},
{0xe28098, 0xa1c6},
{0xe28099, 0xa1c7},
{0xe2809c, 0xa1c8},
***************
*** 399,428 ****
{0xe280b3, 0xa1ed},
{0xe280bb, 0xa2a8},
{0xe28483, 0xa1ee},
! {0xe28496, 0xade2},
! {0xe284a1, 0xade4},
{0xe284a2, 0x8fa2ef},
{0xe284ab, 0xa2f2},
- {0xe285a0, 0xadb5},
- {0xe285a1, 0xadb6},
- {0xe285a2, 0xadb7},
- {0xe285a3, 0xadb8},
- {0xe285a4, 0xadb9},
- {0xe285a5, 0xadba},
- {0xe285a6, 0xadbb},
- {0xe285a7, 0xadbc},
- {0xe285a8, 0xadbd},
- {0xe285a9, 0xadbe},
- {0xe285b0, 0x8ff3f3},
- {0xe285b1, 0x8ff3f4},
- {0xe285b2, 0x8ff3f5},
- {0xe285b3, 0x8ff3f6},
- {0xe285b4, 0x8ff3f7},
- {0xe285b5, 0x8ff3f8},
- {0xe285b6, 0x8ff3f9},
- {0xe285b7, 0x8ff3fa},
- {0xe285b8, 0x8ff3fb},
- {0xe285b9, 0x8ff3fc},
{0xe28690, 0xa2ab},
{0xe28691, 0xa2ac},
{0xe28692, 0xa2aa},
--- 405,413 ----
{0xe280b3, 0xa1ed},
{0xe280bb, 0xa2a8},
{0xe28483, 0xa1ee},
! {0xe28496, 0x8fa2f1},
{0xe284a2, 0x8fa2ef},
{0xe284ab, 0xa2f2},
{0xe28690, 0xa2ab},
{0xe28691, 0xa2ac},
{0xe28692, 0xa2aa},
***************
*** 435,454 ****
{0xe28887, 0xa2e0},
{0xe28888, 0xa2ba},
{0xe2888b, 0xa2bb},
! {0xe28891, 0xadf4},
{0xe2889a, 0xa2e5},
{0xe2889d, 0xa2e7},
{0xe2889e, 0xa1e7},
- {0xe2889f, 0xadf8},
{0xe288a0, 0xa2dc},
- {0xe288a5, 0xa1c2},
{0xe288a7, 0xa2ca},
{0xe288a8, 0xa2cb},
{0xe288a9, 0xa2c1},
{0xe288aa, 0xa2c0},
{0xe288ab, 0xa2e9},
{0xe288ac, 0xa2ea},
- {0xe288ae, 0xadf3},
{0xe288b4, 0xa1e8},
{0xe288b5, 0xa2e8},
{0xe288bd, 0xa2e6},
--- 420,436 ----
{0xe28887, 0xa2e0},
{0xe28888, 0xa2ba},
{0xe2888b, 0xa2bb},
! {0xe28892, 0xa1dd},
{0xe2889a, 0xa2e5},
{0xe2889d, 0xa2e7},
{0xe2889e, 0xa1e7},
{0xe288a0, 0xa2dc},
{0xe288a7, 0xa2ca},
{0xe288a8, 0xa2cb},
{0xe288a9, 0xa2c1},
{0xe288aa, 0xa2c0},
{0xe288ab, 0xa2e9},
{0xe288ac, 0xa2ea},
{0xe288b4, 0xa1e8},
{0xe288b5, 0xa2e8},
{0xe288bd, 0xa2e6},
***************
*** 464,491 ****
{0xe28a86, 0xa2bc},
{0xe28a87, 0xa2bd},
{0xe28aa5, 0xa2dd},
- {0xe28abf, 0xadf9},
{0xe28c92, 0xa2de},
- {0xe291a0, 0xada1},
- {0xe291a1, 0xada2},
- {0xe291a2, 0xada3},
- {0xe291a3, 0xada4},
- {0xe291a4, 0xada5},
- {0xe291a5, 0xada6},
- {0xe291a6, 0xada7},
- {0xe291a7, 0xada8},
- {0xe291a8, 0xada9},
- {0xe291a9, 0xadaa},
- {0xe291aa, 0xadab},
- {0xe291ab, 0xadac},
- {0xe291ac, 0xadad},
- {0xe291ad, 0xadae},
- {0xe291ae, 0xadaf},
- {0xe291af, 0xadb0},
- {0xe291b0, 0xadb1},
- {0xe291b1, 0xadb2},
- {0xe291b2, 0xadb3},
- {0xe291b3, 0xadb4},
{0xe29480, 0xa8a1},
{0xe29481, 0xa8ac},
{0xe29482, 0xa8a2},
--- 446,452 ----
***************
*** 558,565 ****
{0xe38093, 0xa2ae},
{0xe38094, 0xa1cc},
{0xe38095, 0xa1cd},
! {0xe3809d, 0xade0},
! {0xe3809f, 0xade1},
{0xe38181, 0xa4a1},
{0xe38182, 0xa4a2},
{0xe38183, 0xa4a3},
--- 519,525 ----
{0xe38093, 0xa2ae},
{0xe38094, 0xa1cc},
{0xe38095, 0xa1cd},
! {0xe3809c, 0xa1c1},
{0xe38181, 0xa4a1},
{0xe38182, 0xa4a2},
{0xe38183, 0xa4a3},
***************
*** 737,778 ****
{0xe383bc, 0xa1bc},
{0xe383bd, 0xa1b3},
{0xe383be, 0xa1b4},
- {0xe388b1, 0xadea},
- {0xe388b2, 0xadeb},
- {0xe388b9, 0xadec},
- {0xe38aa4, 0xade5},
- {0xe38aa5, 0xade6},
- {0xe38aa6, 0xade7},
- {0xe38aa7, 0xade8},
- {0xe38aa8, 0xade9},
- {0xe38c83, 0xadc6},
- {0xe38c8d, 0xadca},
- {0xe38c94, 0xadc1},
- {0xe38c98, 0xadc4},
- {0xe38ca2, 0xadc2},
- {0xe38ca3, 0xadcc},
- {0xe38ca6, 0xadcb},
- {0xe38ca7, 0xadc5},
- {0xe38cab, 0xadcd},
- {0xe38cb6, 0xadc7},
- {0xe38cbb, 0xadcf},
- {0xe38d89, 0xadc0},
- {0xe38d8a, 0xadce},
- {0xe38d8d, 0xadc3},
- {0xe38d91, 0xadc8},
- {0xe38d97, 0xadc9},
- {0xe38dbb, 0xaddf},
- {0xe38dbc, 0xadef},
- {0xe38dbd, 0xadee},
- {0xe38dbe, 0xaded},
- {0xe38e8e, 0xadd3},
- {0xe38e8f, 0xadd4},
- {0xe38e9c, 0xadd0},
- {0xe38e9d, 0xadd1},
- {0xe38e9e, 0xadd2},
- {0xe38ea1, 0xadd6},
- {0xe38f84, 0xadd5},
- {0xe38f8d, 0xade3},
{0xe4b880, 0xb0ec},
{0xe4b881, 0xc3fa},
{0xe4b882, 0x8fb0a1},
--- 697,702 ----
***************
*** 934,940 ****
{0xe4bbb6, 0xb7ef},
{0xe4bbb7, 0xd0c3},
{0xe4bbbb, 0xc7a4},
- {0xe4bbbc, 0x8ff4af},
{0xe4bbbd, 0x8fb0cf},
{0xe4bbbe, 0x8fb0d0},
{0xe4bbbf, 0x8fb0d1},
--- 858,863 ----
***************
*** 1281,1287 ****
{0xe583af, 0x8fb2c4},
{0xe583b1, 0x8fb2c5},
{0xe583b2, 0x8fb2d1},
- {0xe583b4, 0x8ff4b0},
{0xe583b5, 0xd1ac},
{0xe583b6, 0x8fb2c6},
{0xe583b9, 0xd1ab},
--- 1204,1209 ----
***************
*** 1457,1463 ****
{0xe587a7, 0xc2fc},
{0xe587a9, 0xd1de},
{0xe587aa, 0xc6e4},
- {0xe587ac, 0x8ff4b1},
{0xe587ad, 0xd1df},
{0xe587ae, 0x8fb3ab},
{0xe587b0, 0xd1e0},
--- 1379,1384 ----
***************
*** 1658,1664 ****
{0xe58c83, 0x8fb3fc},
{0xe58c85, 0xcaf1},
{0xe58c86, 0xd2b2},
- {0xe58c87, 0x8ff4b2},
{0xe58c88, 0xd2b3},
{0xe58c8a, 0x8fb3fd},
{0xe58c8b, 0x8fb3fe},
--- 1579,1584 ----
***************
*** 1682,1688 ****
{0xe58ca0, 0xbea2},
{0xe58ca1, 0xb6a9},
{0xe58ca3, 0xd2ba},
- {0xe58ca4, 0x8ff4b3},
{0xe58ca5, 0x8fb4a9},
{0xe58ca7, 0x8fb4aa},
{0xe58ca8, 0x8fb4ab},
--- 1602,1607 ----
***************
*** 1922,1928 ****
{0xe59286, 0xd2f6},
{0xe59288, 0x8fb5a4},
{0xe59289, 0x8fb5a5},
- {0xe5928a, 0x8ff4b5},
{0xe5928b, 0xbaf0},
{0xe5928c, 0xcfc2},
{0xe5928d, 0x8fb5a6},
--- 1841,1846 ----
***************
*** 2334,2340 ****
{0xe59d90, 0xbac1},
{0xe59d91, 0xb9a3},
{0xe59d92, 0x8fb7c0},
- {0xe59d99, 0x8ff4b6},
{0xe59da1, 0xd4b3},
{0xe59da2, 0x8fb7c1},
{0xe59da4, 0xbaa5},
--- 2252,2257 ----
***************
*** 2516,2522 ****
{0xe5a297, 0xc1fd},
{0xe5a29c, 0xc4c6},
{0xe5a29d, 0x8fb8c7},
- {0xe5a29e, 0x8ff4b9},
{0xe5a29f, 0xd4d2},
{0xe5a2a0, 0x8fb8c8},
{0xe5a2a1, 0x8fb8c9},
--- 2433,2438 ----
***************
*** 3001,3007 ****
{0xe5afa8, 0xdccd},
{0xe5afa9, 0xbfb3},
{0xe5afab, 0xd5ed},
- {0xe5afac, 0x8ff4ba},
{0xe5afae, 0xcec0},
{0xe5afaf, 0x8fbae5},
{0xe5afb0, 0xd5ee},
--- 2917,2922 ----
***************
*** 3154,3160 ****
{0xe5b3b1, 0x8fbbbf},
{0xe5b3b2, 0x8fbbc0},
{0xe5b3b4, 0x8fbbc1},
- {0xe5b3b5, 0x8ff4bb},
{0xe5b3b6, 0xc5e7},
{0xe5b3ba, 0xd6b8},
{0xe5b3bb, 0xbdd4},
--- 3069,3074 ----
***************
*** 3199,3205 ****
{0xe5b590, 0xcdf2},
{0xe5b591, 0x8fbbd5},
{0xe5b592, 0xd6c9},
- {0xe5b593, 0x8ff4bc},
{0xe5b595, 0x8fbbd4},
{0xe5b599, 0x8fbbd6},
{0xe5b59c, 0xd6bf},
--- 3113,3118 ----
***************
*** 3592,3598 ****
{0xe5beb1, 0x8fbdb6},
{0xe5beb3, 0xc6c1},
{0xe5beb4, 0xc4a7},
- {0xe5beb7, 0x8ff4be},
{0xe5beb8, 0x8fbdb7},
{0xe5beb9, 0xc5b0},
{0xe5bebc, 0xd7d5},
--- 3505,3510 ----
***************
*** 3724,3730 ****
{0xe68282, 0x8fbded},
{0xe68283, 0xd7fd},
{0xe68284, 0xd8a1},
- {0xe68285, 0x8ff4bf},
{0xe68286, 0x8fbdee},
{0xe68288, 0x8fbdef},
{0xe68289, 0xbcbd},
--- 3636,3641 ----
***************
*** 3840,3846 ****
{0xe6849c, 0x8fbec4},
{0xe6849e, 0x8fbec5},
{0xe6849f, 0xb4b6},
- {0xe684a0, 0x8ff4c0},
{0xe684a1, 0xd8bb},
{0xe684a2, 0x8fbec6},
{0xe684a7, 0xd8c3},
--- 3751,3756 ----
***************
*** 4494,4500 ****
{0xe69588, 0xdac3},
{0xe69589, 0x8fc1cc},
{0xe6958d, 0xdac6},
- {0xe6958e, 0x8ff4c1},
{0xe6958f, 0xc9d2},
{0xe69590, 0x8fc1cd},
{0xe69591, 0xb5df},
--- 4404,4409 ----
***************
*** 4653,4659 ****
{0xe698b6, 0xdae4},
{0xe698b7, 0x8fc2b0},
{0xe698b9, 0x8fc2af},
- {0xe698bb, 0x8ff4c2},
{0xe698bc, 0xc3eb},
{0xe698bf, 0xdba6},
{0xe69980, 0x8fc2b1},
--- 4562,4567 ----
***************
*** 4683,4689 ****
{0xe699a1, 0x8fc2be},
{0xe699a2, 0xdaf1},
{0xe699a4, 0xdaed},
- {0xe699a5, 0x8ff4c3},
{0xe699a6, 0xb3a2},
{0xe699a7, 0xdaee},
{0xe699a8, 0xdaef},
--- 4591,4596 ----
***************
*** 4956,4962 ****
{0xe69fbb, 0xbaba},
{0xe69fbe, 0xcbef},
{0xe69fbf, 0xb3c1},
- {0xe6a081, 0x8ff4c6},
{0xe6a082, 0xc4ce},
{0xe6a083, 0xc6ca},
{0xe6a084, 0xb1c9},
--- 4863,4868 ----
***************
*** 5344,5350 ****
{0xe6a9a6, 0xdcf5},
{0xe6a9a7, 0x8fc5b2},
{0xe6a9aa, 0x8fc5b3},
- {0xe6a9ab, 0x8ff4c9},
{0xe6a9b1, 0x8fc5b4},
{0xe6a9b2, 0xdcef},
{0xe6a9b3, 0x8fc5b5},
--- 5250,5255 ----
***************
*** 5406,5412 ****
{0xe6ab9d, 0x8fc5d4},
{0xe6ab9e, 0xdda9},
{0xe6ab9f, 0xddab},
- {0xe6aba2, 0x8ff4ca},
{0xe6aba4, 0x8fc5d5},
{0xe6aba7, 0x8fc5d6},
{0xe6aba8, 0xc8a7},
--- 5311,5316 ----
***************
*** 5854,5866 ****
{0xe6b7af, 0x8fc7e2},
{0xe6b7b0, 0x8fc7e3},
{0xe6b7b1, 0xbfbc},
- {0xe6b7b2, 0x8ff4cc},
{0xe6b7b3, 0xbddf},
{0xe6b7b4, 0x8fc7e4},
{0xe6b7b5, 0xcaa5},
{0xe6b7b6, 0x8fc7e5},
{0xe6b7b7, 0xbaae},
- {0xe6b7b8, 0x8ff4cb},
{0xe6b7b9, 0xdebb},
{0xe6b7ba, 0xdec9},
{0xe6b7bb, 0xc5ba},
--- 5758,5768 ----
***************
*** 6170,6176 ****
{0xe780a3, 0x8fc9b1},
{0xe780a6, 0xc3f5},
{0xe780a7, 0xc2ed},
- {0xe780a8, 0x8ff4cd},
{0xe780ac, 0xc0a5},
{0xe780af, 0x8fc9b2},
{0xe780b0, 0xdfd0},
--- 6072,6077 ----
***************
*** 6236,6242 ****
{0xe782b8, 0xdfda},
{0xe782b9, 0xc5c0},
{0xe782ba, 0xb0d9},
- {0xe782bb, 0x8ff4ae},
{0xe78388, 0xcef5},
{0xe7838a, 0x8fc9d8},
{0xe7838b, 0xdfde},
--- 6137,6142 ----
***************
*** 6810,6816 ****
{0xe793bb, 0x8fccfb},
{0xe793bc, 0x8fccfc},
{0xe793bf, 0x8fccfd},
- {0xe79481, 0x8ff4d0},
{0xe79483, 0xe1ac},
{0xe79484, 0xe1ab},
{0xe79485, 0xe1ad},
--- 6710,6715 ----
***************
*** 7075,7081 ****
{0xe799be, 0xc9b4},
{0xe79a80, 0xe2a5},
{0xe79a81, 0x8fceb5},
- {0xe79a82, 0x8ff4d1},
{0xe79a83, 0xe2a6},
{0xe79a84, 0xc5aa},
{0xe79a85, 0x8fceb6},
--- 6974,6979 ----
***************
*** 7095,7101 ****
{0xe79a9b, 0x8fceba},
{0xe79a9c, 0x8fcebb},
{0xe79a9d, 0x8fcebc},
- {0xe79a9e, 0x8ff4d2},
{0xe79a9f, 0x8fcebd},
{0xe79aa0, 0x8fcebe},
{0xe79aa2, 0x8fcebf},
--- 6993,6998 ----
***************
*** 7458,7464 ****
{0xe7a4ab, 0xe3aa},
{0xe7a4ac, 0xe3a9},
{0xe7a4ad, 0x8fd0c7},
- {0xe7a4b0, 0x8ff4d4},
{0xe7a4b1, 0x8fd0c8},
{0xe7a4b4, 0x8fd0c9},
{0xe7a4b5, 0x8fd0ca},
--- 7355,7360 ----
***************
*** 7736,7742 ****
{0xe7aba3, 0xbdd7},
{0xe7aba5, 0xc6b8},
{0xe7aba6, 0xe3f0},
- {0xe7aba7, 0x8ff4d9},
{0xe7aba8, 0x8fd1ea},
{0xe7aba9, 0x8fd1eb},
{0xe7abaa, 0xc3a8},
--- 7632,7637 ----
***************
*** 8191,8197 ****
{0xe7b69d, 0x8fd3fe},
{0xe7b69e, 0x8fd4a1},
{0xe7b69f, 0xe5c6},
- {0xe7b6a0, 0x8ff4dc},
{0xe7b6a2, 0xe5c2},
{0xe7b6a3, 0xe5bc},
{0xe7b6a6, 0x8fd4a2},
--- 8086,8091 ----
***************
*** 8232,8238 ****
{0xe7b791, 0xced0},
{0xe7b792, 0xbdef},
{0xe7b795, 0xe5ee},
- {0xe7b796, 0x8ff4dd},
{0xe7b797, 0x8fd4b1},
{0xe7b798, 0xe5c8},
{0xe7b799, 0x8fd4b2},
--- 8126,8131 ----
***************
*** 9034,9040 ****
{0xe88d96, 0x8fd7fb},
{0xe88d97, 0x8fd7fc},
{0xe88d98, 0xc1f1},
- {0xe88da2, 0x8ff4df},
{0xe88db0, 0x8fd7fd},
{0xe88db3, 0xe8ba},
{0xe88db5, 0xe8bb},
--- 8927,8932 ----
***************
*** 9372,9378 ****
{0xe896ad, 0x8fd9f4},
{0xe896ae, 0xccf9},
{0xe896af, 0xbdf2},
- {0xe896b0, 0x8ff4e1},
{0xe896b4, 0x8fd9e7},
{0xe896b6, 0x8fd9e8},
{0xe896b7, 0x8fd9e9},
--- 9264,9269 ----
***************
*** 9711,9717 ****
{0xe8a083, 0x8fdbe9},
{0xe8a085, 0xeaa4},
{0xe8a086, 0x8fdbea},
- {0xe8a087, 0x8ff4e4},
{0xe8a089, 0x8fdbeb},
{0xe8a08a, 0x8fdbec},
{0xe8a08b, 0x8fdbed},
--- 9602,9607 ----
***************
*** 10276,10282 ****
{0xe8adbc, 0x8fded5},
{0xe8adbd, 0xeca5},
{0xe8adbe, 0x8fded6},
- {0xe8adbf, 0x8ff4e6},
{0xe8ae80, 0xeca6},
{0xe8ae81, 0x8fded7},
{0xe8ae83, 0xbbbe},
--- 10166,10171 ----
***************
*** 10432,10438 ****
{0xe8b3af, 0x8fdfb8},
{0xe8b3b0, 0x8fdfb9},
{0xe8b3b2, 0x8fdfba},
- {0xe8b3b4, 0x8ff4e7},
{0xe8b3b5, 0x8fdfbb},
{0xe8b3b7, 0x8fdfbc},
{0xe8b3b8, 0x8fdfbd},
--- 10321,10326 ----
***************
*** 10473,10479 ****
{0xe8b5b1, 0xece1},
{0xe8b5b3, 0xece2},
{0xe8b5b4, 0xc9eb},
- {0xe8b5b6, 0x8ff4e8},
{0xe8b5b7, 0xb5af},
{0xe8b5bf, 0x8fdfca},
{0xe8b681, 0xece3},
--- 10361,10366 ----
***************
*** 10997,11003 ****
{0xe9839b, 0xeebe},
{0xe9839c, 0x8fe2a9},
{0xe9839d, 0x8fe2aa},
- {0xe9839e, 0x8ff4ec},
{0xe9839f, 0x8fe2ab},
{0xe983a1, 0xb7b4},
{0xe983a2, 0xeebb},
--- 10884,10889 ----
***************
*** 11025,11031 ****
{0xe98490, 0x8fe2bb},
{0xe98492, 0xeec0},
{0xe98494, 0x8fe2bc},
- {0xe98495, 0x8ff4ee},
{0xe98496, 0x8fe2bd},
{0xe98497, 0x8fe2be},
{0xe98498, 0x8fe2bf},
--- 10911,10916 ----
***************
*** 11599,11605 ****
{0xe9968e, 0x8fe6a6},
{0xe9968f, 0xb1bc},
{0xe99691, 0xb4d7},
- {0xe99692, 0x8ff4f1},
{0xe99693, 0xb4d6},
{0xe99694, 0xefdc},
{0xe99696, 0xefdd},
--- 11484,11489 ----
***************
*** 11848,11854 ****
{0xe99cb3, 0x8fe7ac},
{0xe99cb8, 0xdbb1},
{0xe99cb9, 0xf0c8},
- {0xe99cbb, 0x8ff4f4},
{0xe99cbd, 0xf0c9},
{0xe99cbe, 0xf0ca},
{0xe99d81, 0x8fe7ad},
--- 11732,11737 ----
***************
*** 11859,11868 ****
{0xe99d88, 0xf0cd},
{0xe99d89, 0xf0cf},
{0xe99d8a, 0x8fe7af},
- {0xe99d8d, 0x8ff4f5},
{0xe99d8e, 0x8fe7b0},
{0xe99d8f, 0x8fe7b1},
- {0xe99d91, 0x8ff4f6},
{0xe99d92, 0xc0c4},
{0xe99d95, 0x8fe7b2},
{0xe99d96, 0xccf7},
--- 11742,11749 ----
***************
*** 12178,12184 ****
{0xe9a699, 0xb9e1},
{0xe9a69b, 0x8fe8f7},
{0xe9a69d, 0x8fe8f8},
- {0xe9a69e, 0x8ff4fa},
{0xe9a69f, 0x8fe8f9},
{0xe9a6a5, 0xf1c6},
{0xe9a6a6, 0x8fe8fa},
--- 12059,12064 ----
***************
*** 12329,12335 ****
{0xe9ab96, 0x8fe9ea},
{0xe9ab97, 0x8fe9eb},
{0xe9ab98, 0xb9e2},
- {0xe9ab99, 0x8ff4fb},
{0xe9ab9b, 0x8fe9ec},
{0xe9ab9c, 0x8fe9ed},
{0xe9ab9e, 0xf1f4},
--- 12209,12214 ----
***************
*** 12436,12442 ****
{0xe9adad, 0x8feaca},
{0xe9adae, 0x8feacb},
{0xe9adaf, 0xcfa5},
- {0xe9adb2, 0x8ff4fc},
{0xe9adb3, 0x8feacc},
{0xe9adb4, 0xf2b7},
{0xe9adb5, 0x8feacd},
--- 12315,12320 ----
***************
*** 12846,12852 ****
{0xe9bb8e, 0xf3d5},
{0xe9bb8f, 0xf3d6},
{0xe9bb90, 0xf3d7},
- {0xe9bb91, 0x8ff4fe},
{0xe9bb92, 0xb9f5},
{0xe9bb94, 0xf3d8},
{0xe9bb95, 0x8fecf4},
--- 12724,12729 ----
***************
*** 12977,13029 ****
{0xe9bea2, 0x8fede1},
{0xe9bea3, 0x8fede2},
{0xe9bea5, 0x8fede3},
- {0xefa4a9, 0x8ff4c5},
- {0xefa79c, 0x8ff4f2},
- {0xefa88e, 0x8ff4b4},
- {0xefa88f, 0x8ff4b7},
- {0xefa890, 0x8ff4b8},
- {0xefa891, 0x8ff4bd},
- {0xefa892, 0x8ff4c4},
- {0xefa893, 0x8ff4c7},
- {0xefa894, 0x8ff4c8},
- {0xefa895, 0x8ff4ce},
- {0xefa896, 0x8ff4cf},
- {0xefa897, 0x8ff4d3},
- {0xefa898, 0x8ff4d5},
- {0xefa899, 0x8ff4d6},
- {0xefa89a, 0x8ff4d7},
- {0xefa89b, 0x8ff4d8},
- {0xefa89c, 0x8ff4da},
- {0xefa89d, 0x8ff4db},
- {0xefa89e, 0x8ff4de},
- {0xefa89f, 0x8ff4e0},
- {0xefa8a0, 0x8ff4e2},
- {0xefa8a1, 0x8ff4e3},
- {0xefa8a2, 0x8ff4e5},
- {0xefa8a3, 0x8ff4e9},
- {0xefa8a4, 0x8ff4ea},
- {0xefa8a5, 0x8ff4eb},
- {0xefa8a6, 0x8ff4ed},
- {0xefa8a7, 0x8ff4ef},
- {0xefa8a8, 0x8ff4f0},
- {0xefa8a9, 0x8ff4f3},
- {0xefa8aa, 0x8ff4f7},
- {0xefa8ab, 0x8ff4f8},
- {0xefa8ac, 0x8ff4f9},
- {0xefa8ad, 0x8ff4fd},
{0xefbc81, 0xa1aa},
- {0xefbc82, 0x8ff4aa},
{0xefbc83, 0xa1f4},
{0xefbc84, 0xa1f0},
{0xefbc85, 0xa1f3},
{0xefbc86, 0xa1f5},
- {0xefbc87, 0x8ff4a9},
{0xefbc88, 0xa1ca},
{0xefbc89, 0xa1cb},
{0xefbc8a, 0xa1f6},
{0xefbc8b, 0xa1dc},
{0xefbc8c, 0xa1a4},
- {0xefbc8d, 0xa1dd},
{0xefbc8e, 0xa1a5},
{0xefbc8f, 0xa1bf},
{0xefbc90, 0xa3b0},
--- 12854,12869 ----
***************
*** 13070,13076 ****
{0xefbcb9, 0xa3d9},
{0xefbcba, 0xa3da},
{0xefbcbb, 0xa1ce},
- {0xefbcbc, 0xa1c0},
{0xefbcbd, 0xa1cf},
{0xefbcbe, 0xa1b0},
{0xefbcbf, 0xa1b2},
--- 12910,12915 ----
***************
*** 13104,13110 ****
{0xefbd9b, 0xa1d0},
{0xefbd9c, 0xa1c3},
{0xefbd9d, 0xa1d1},
- {0xefbd9e, 0xa1c1},
{0xefbda1, 0x8ea1},
{0xefbda2, 0x8ea2},
{0xefbda3, 0x8ea3},
--- 12943,12948 ----
***************
*** 13168,13177 ****
{0xefbe9d, 0x8edd},
{0xefbe9e, 0x8ede},
{0xefbe9f, 0x8edf},
- {0xefbfa0, 0xa1f1},
- {0xefbfa1, 0xa1f2},
- {0xefbfa2, 0xa2cc},
{0xefbfa3, 0xa1b1},
- {0xefbfa4, 0x8fa2c3},
{0xefbfa5, 0xa1ef}
};
--- 13006,13011 ----
*** utf8_to_euc_kr.map.orig Fri May 15 17:56:06 2015
--- utf8_to_euc_kr.map Fri Nov 27 16:25:41 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapEUC_KR[ 8227 ] = {
{0xc2a1, 0xa2ae},
{0xc2a4, 0xa2b4},
{0xc2a7, 0xa1d7},
{0xc2a8, 0xa1a7},
{0xc2aa, 0xa8a3},
{0xc2ad, 0xa1a9},
- {0xc2ae, 0xa2e7},
{0xc2b0, 0xa1c6},
{0xc2b1, 0xa1be},
{0xc2b2, 0xa9f7},
--- 1,12 ----
! /* src/backend/utils/mb/Unicode/utf8_to_euc_kr.map */
!
! static const pg_utf_to_local ULmapEUC_KR[ 8224 ] = {
{0xc2a1, 0xa2ae},
{0xc2a4, 0xa2b4},
{0xc2a7, 0xa1d7},
{0xc2a8, 0xa1a7},
{0xc2aa, 0xa8a3},
{0xc2ad, 0xa1a9},
{0xc2b0, 0xa1c6},
{0xc2b1, 0xa1be},
{0xc2b2, 0xa9f7},
***************
*** 189,195 ****
{0xe28282, 0xa9fc},
{0xe28283, 0xa9fd},
{0xe28284, 0xa9fe},
- {0xe282ac, 0xa2e6},
{0xe28483, 0xa1c9},
{0xe28489, 0xa2b5},
{0xe28493, 0xa7a4},
--- 190,195 ----
***************
*** 806,812 ****
{0xe389b9, 0xa8ca},
{0xe389ba, 0xa8cb},
{0xe389bb, 0xa8cc},
- {0xe389be, 0xa2e8},
{0xe389bf, 0xa2de},
{0xe38e80, 0xa7c9},
{0xe38e81, 0xa7ca},
--- 806,811 ----
*** utf8_to_johab.map.orig Fri May 15 17:56:06 2015
--- utf8_to_johab.map Fri Nov 27 16:26:10 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapJOHAB[ 17049 ] = {
{0xc2a1, 0xd9ae},
{0xc2a4, 0xd9b4},
{0xc2a7, 0xd967},
{0xc2a8, 0xd937},
{0xc2aa, 0xdca3},
{0xc2ad, 0xd939},
- {0xc2ae, 0xd9e7},
{0xc2b0, 0xd956},
{0xc2b1, 0xd94e},
{0xc2b2, 0xdd99},
--- 1,12 ----
! /* src/backend/utils/mb/Unicode/utf8_to_johab.map */
!
! static const pg_utf_to_local ULmapJOHAB[ 17046 ] = {
{0xc2a1, 0xd9ae},
{0xc2a4, 0xd9b4},
{0xc2a7, 0xd967},
{0xc2a8, 0xd937},
{0xc2aa, 0xdca3},
{0xc2ad, 0xd939},
{0xc2b0, 0xd956},
{0xc2b1, 0xd94e},
{0xc2b2, 0xdd99},
***************
*** 189,195 ****
{0xe28282, 0xdd9e},
{0xe28283, 0xdd9f},
{0xe28284, 0xdda0},
- {0xe282ac, 0xd9e6},
{0xe28483, 0xd959},
{0xe28489, 0xd9b5},
{0xe28493, 0xdc34},
--- 190,195 ----
***************
*** 806,812 ****
{0xe389b9, 0xdcca},
{0xe389ba, 0xdccb},
{0xe389bb, 0xdccc},
- {0xe389be, 0xd9e8},
{0xe389bf, 0xd9de},
{0xe38e80, 0xdc59},
{0xe38e81, 0xdc5a},
--- 806,811 ----
*** utf8_to_sjis.map.orig Fri May 15 17:56:06 2015
--- utf8_to_sjis.map Fri Nov 27 16:25:56 2015
***************
*** 1,11 ****
! static const pg_utf_to_local ULmapSJIS[ 7398 ] = {
! {0xc19c, 0x815f},
! {0xc2a2, 0x8191},
! {0xc2a3, 0x8192},
! {0xc2a5, 0x5c},
{0xc2a7, 0x8198},
{0xc2a8, 0x814e},
- {0xc2ac, 0x81ca},
{0xc2b0, 0x818b},
{0xc2b1, 0x817d},
{0xc2b4, 0x814c},
--- 1,8 ----
! /* src/backend/utils/mb/Unicode/utf8_to_sjis.map */
!
! static const pg_utf_to_local ULmapSJIS[ 7389 ] = {
{0xc2a7, 0x8198},
{0xc2a8, 0x814e},
{0xc2b0, 0x818b},
{0xc2b1, 0x817d},
{0xc2b4, 0x814c},
***************
*** 128,134 ****
{0xd191, 0x8476},
{0xe28090, 0x815d},
{0xe28095, 0x815c},
- {0xe28096, 0x8161},
{0xe28098, 0x8165},
{0xe28099, 0x8166},
{0xe2809c, 0x8167},
--- 125,130 ----
***************
*** 141,147 ****
{0xe280b2, 0x818c},
{0xe280b3, 0x818d},
{0xe280bb, 0x81a6},
- {0xe280be, 0x7e},
{0xe28483, 0x818e},
{0xe28496, 0xfa59},
{0xe284a1, 0xfa5a},
--- 137,142 ----
***************
*** 179,185 ****
{0xe28888, 0x81b8},
{0xe2888b, 0x81b9},
{0xe28891, 0x8794},
- {0xe28892, 0x817c},
{0xe2889a, 0x81e3},
{0xe2889d, 0x81e5},
{0xe2889e, 0x8187},
--- 174,179 ----
***************
*** 302,308 ****
{0xe38093, 0x81ac},
{0xe38094, 0x816b},
{0xe38095, 0x816c},
- {0xe3809c, 0x8160},
{0xe3809d, 0x8780},
{0xe3809f, 0x8781},
{0xe38181, 0x829f},
--- 296,301 ----
*** utf8_to_uhc.map.orig Fri May 15 17:56:06 2015
--- utf8_to_uhc.map Fri Nov 27 16:26:11 2015
***************
*** 1,4 ****
! static const pg_utf_to_local ULmapUHC[ 17237 ] = {
{0xc2a1, 0xa2ae},
{0xc2a4, 0xa2b4},
{0xc2a7, 0xa1d7},
--- 1,6 ----
! /* src/backend/utils/mb/Unicode/utf8_to_uhc.map */
!
! static const pg_utf_to_local ULmapUHC[ 17048 ] = {
{0xc2a1, 0xa2ae},
{0xc2a4, 0xa2b4},
{0xc2a7, 0xa1d7},
***************
*** 806,812 ****
{0xe389b9, 0xa8ca},
{0xe389ba, 0xa8cb},
{0xe389bb, 0xa8cc},
- {0xe389be, 0xa2e8},
{0xe389bf, 0xa2de},
{0xe38e80, 0xa7c9},
{0xe38e81, 0xa7ca},
--- 808,813 ----
***************
*** 16680,16873 ****
{0xed9ea1, 0xc650},
{0xed9ea2, 0xc651},
{0xed9ea3, 0xc652},
- {0xee8080, 0xc9a1},
- {0xee8081, 0xc9a2},
- {0xee8082, 0xc9a3},
- {0xee8083, 0xc9a4},
- {0xee8084, 0xc9a5},
- {0xee8085, 0xc9a6},
- {0xee8086, 0xc9a7},
- {0xee8087, 0xc9a8},
- {0xee8088, 0xc9a9},
- {0xee8089, 0xc9aa},
- {0xee808a, 0xc9ab},
- {0xee808b, 0xc9ac},
- {0xee808c, 0xc9ad},
- {0xee808d, 0xc9ae},
- {0xee808e, 0xc9af},
- {0xee808f, 0xc9b0},
- {0xee8090, 0xc9b1},
- {0xee8091, 0xc9b2},
- {0xee8092, 0xc9b3},
- {0xee8093, 0xc9b4},
- {0xee8094, 0xc9b5},
- {0xee8095, 0xc9b6},
- {0xee8096, 0xc9b7},
- {0xee8097, 0xc9b8},
- {0xee8098, 0xc9b9},
- {0xee8099, 0xc9ba},
- {0xee809a, 0xc9bb},
- {0xee809b, 0xc9bc},
- {0xee809c, 0xc9bd},
- {0xee809d, 0xc9be},
- {0xee809e, 0xc9bf},
- {0xee809f, 0xc9c0},
- {0xee80a0, 0xc9c1},
- {0xee80a1, 0xc9c2},
- {0xee80a2, 0xc9c3},
- {0xee80a3, 0xc9c4},
- {0xee80a4, 0xc9c5},
- {0xee80a5, 0xc9c6},
- {0xee80a6, 0xc9c7},
- {0xee80a7, 0xc9c8},
- {0xee80a8, 0xc9c9},
- {0xee80a9, 0xc9ca},
- {0xee80aa, 0xc9cb},
- {0xee80ab, 0xc9cc},
- {0xee80ac, 0xc9cd},
- {0xee80ad, 0xc9ce},
- {0xee80ae, 0xc9cf},
- {0xee80af, 0xc9d0},
- {0xee80b0, 0xc9d1},
- {0xee80b1, 0xc9d2},
- {0xee80b2, 0xc9d3},
- {0xee80b3, 0xc9d4},
- {0xee80b4, 0xc9d5},
- {0xee80b5, 0xc9d6},
- {0xee80b6, 0xc9d7},
- {0xee80b7, 0xc9d8},
- {0xee80b8, 0xc9d9},
- {0xee80b9, 0xc9da},
- {0xee80ba, 0xc9db},
- {0xee80bb, 0xc9dc},
- {0xee80bc, 0xc9dd},
- {0xee80bd, 0xc9de},
- {0xee80be, 0xc9df},
- {0xee80bf, 0xc9e0},
- {0xee8180, 0xc9e1},
- {0xee8181, 0xc9e2},
- {0xee8182, 0xc9e3},
- {0xee8183, 0xc9e4},
- {0xee8184, 0xc9e5},
- {0xee8185, 0xc9e6},
- {0xee8186, 0xc9e7},
- {0xee8187, 0xc9e8},
- {0xee8188, 0xc9e9},
- {0xee8189, 0xc9ea},
- {0xee818a, 0xc9eb},
- {0xee818b, 0xc9ec},
- {0xee818c, 0xc9ed},
- {0xee818d, 0xc9ee},
- {0xee818e, 0xc9ef},
- {0xee818f, 0xc9f0},
- {0xee8190, 0xc9f1},
- {0xee8191, 0xc9f2},
- {0xee8192, 0xc9f3},
- {0xee8193, 0xc9f4},
- {0xee8194, 0xc9f5},
- {0xee8195, 0xc9f6},
- {0xee8196, 0xc9f7},
- {0xee8197, 0xc9f8},
- {0xee8198, 0xc9f9},
- {0xee8199, 0xc9fa},
- {0xee819a, 0xc9fb},
- {0xee819b, 0xc9fc},
- {0xee819c, 0xc9fd},
- {0xee819d, 0xc9fe},
- {0xee819e, 0xfea1},
- {0xee819f, 0xfea2},
- {0xee81a0, 0xfea3},
- {0xee81a1, 0xfea4},
- {0xee81a2, 0xfea5},
- {0xee81a3, 0xfea6},
- {0xee81a4, 0xfea7},
- {0xee81a5, 0xfea8},
- {0xee81a6, 0xfea9},
- {0xee81a7, 0xfeaa},
- {0xee81a8, 0xfeab},
- {0xee81a9, 0xfeac},
- {0xee81aa, 0xfead},
- {0xee81ab, 0xfeae},
- {0xee81ac, 0xfeaf},
- {0xee81ad, 0xfeb0},
- {0xee81ae, 0xfeb1},
- {0xee81af, 0xfeb2},
- {0xee81b0, 0xfeb3},
- {0xee81b1, 0xfeb4},
- {0xee81b2, 0xfeb5},
- {0xee81b3, 0xfeb6},
- {0xee81b4, 0xfeb7},
- {0xee81b5, 0xfeb8},
- {0xee81b6, 0xfeb9},
- {0xee81b7, 0xfeba},
- {0xee81b8, 0xfebb},
- {0xee81b9, 0xfebc},
- {0xee81ba, 0xfebd},
- {0xee81bb, 0xfebe},
- {0xee81bc, 0xfebf},
- {0xee81bd, 0xfec0},
- {0xee81be, 0xfec1},
- {0xee81bf, 0xfec2},
- {0xee8280, 0xfec3},
- {0xee8281, 0xfec4},
- {0xee8282, 0xfec5},
- {0xee8283, 0xfec6},
- {0xee8284, 0xfec7},
- {0xee8285, 0xfec8},
- {0xee8286, 0xfec9},
- {0xee8287, 0xfeca},
- {0xee8288, 0xfecb},
- {0xee8289, 0xfecc},
- {0xee828a, 0xfecd},
- {0xee828b, 0xfece},
- {0xee828c, 0xfecf},
- {0xee828d, 0xfed0},
- {0xee828e, 0xfed1},
- {0xee828f, 0xfed2},
- {0xee8290, 0xfed3},
- {0xee8291, 0xfed4},
- {0xee8292, 0xfed5},
- {0xee8293, 0xfed6},
- {0xee8294, 0xfed7},
- {0xee8295, 0xfed8},
- {0xee8296, 0xfed9},
- {0xee8297, 0xfeda},
- {0xee8298, 0xfedb},
- {0xee8299, 0xfedc},
- {0xee829a, 0xfedd},
- {0xee829b, 0xfede},
- {0xee829c, 0xfedf},
- {0xee829d, 0xfee0},
- {0xee829e, 0xfee1},
- {0xee829f, 0xfee2},
- {0xee82a0, 0xfee3},
- {0xee82a1, 0xfee4},
- {0xee82a2, 0xfee5},
- {0xee82a3, 0xfee6},
- {0xee82a4, 0xfee7},
- {0xee82a5, 0xfee8},
- {0xee82a6, 0xfee9},
- {0xee82a7, 0xfeea},
- {0xee82a8, 0xfeeb},
- {0xee82a9, 0xfeec},
- {0xee82aa, 0xfeed},
- {0xee82ab, 0xfeee},
- {0xee82ac, 0xfeef},
- {0xee82ad, 0xfef0},
- {0xee82ae, 0xfef1},
- {0xee82af, 0xfef2},
- {0xee82b0, 0xfef3},
- {0xee82b1, 0xfef4},
- {0xee82b2, 0xfef5},
- {0xee82b3, 0xfef6},
- {0xee82b4, 0xfef7},
- {0xee82b5, 0xfef8},
- {0xee82b6, 0xfef9},
- {0xee82b7, 0xfefa},
- {0xee82b8, 0xfefb},
- {0xee82b9, 0xfefc},
- {0xee82ba, 0xfefd},
- {0xee82bb, 0xfefe},
{0xefa480, 0xcbd0},
{0xefa481, 0xcbd6},
{0xefa482, 0xcbe7},
--- 16681,16686 ------
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
I wrote:
gb18030_to_utf8.map utf8_to_gb18030.map
Could not find the reference file gb-18030-2000.xml, whose origin is
unstated anyway.
Ah, scratch that complaint; digging in our git history turned up the
origin of that file, so I double-checked it and then updated the script
with a comment. We're good on that conversion ... but the other ones
still have problems.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
I wrote:
I have not attempted to reverify the files in utils/mb/Unicode against the
original Unicode Consortium data, but maybe we ought to do that before
taking any further steps here.I downloaded the mapping files from unicode.org and attempted to verify
that the Unicode/*.map files could be reproduced from the stated sources.
Most of them are okay, but I failed to verify these:euc_cn_to_utf8.map utf8_to_euc_cn.map
Could not find the reference file GB2312.TXT; it is not at unicode.org
gb18030_to_utf8.map utf8_to_gb18030.map
Could not find the reference file gb-18030-2000.xml, whose origin is
unstated anyway.euc_jp_to_utf8.map utf8_to_euc_jp.map
euc_kr_to_utf8.map utf8_to_euc_kr.map
johab_to_utf8.map utf8_to_johab.map
uhc_to_utf8.map utf8_to_uhc.mapThese four all have minor to significant differences from what I got by
running the generation scripts. See attached diffs.utf8_to_sjis.map
It's very disturbing that this fails to verify when its allegedly inverse
file does verify;
either the script is broken or somebody did sloppy
manual editing.
Manual editing.
I explain why the manual editing is necessary.
One of the most famous problems with Unicode is "wave dash"
(U+301C). According the Unicode consortium's Unicode/SJIS map, it
corresponds to 0x8160 of Shift_JIS. Unfortunately this was a mistake
in Unicode (the glyph of Shift_JIS and Unicode is slightly different -
looks like to be rotated in 90 degrees of wave dash in vertical
scripting. Probably they did not understand the Japanese vertical
writing at that time). So later on the Unicode consortium decided to
add another "wave dash" as U+FF5E which has a correct glyph of "wave
dash". However since Unicode already decided that U+301C corresponds
to 0x8160 of Shift_JIS, there's no Shift_JIS code corresponding to
U+FF5E. Unlike Unicode's definition, Microsoft defines that 0x8160
(wave dash) corresponds to U+FF5E. This is widely used in Japan. So I
decided to hire this for "wave dash". i.e.
0x8160 -> U+FF5E (sjis_to_utf8.map)
U+301C -> 0x8160 (utf_to_sjis.map)
U+FF5E -> 0x8160 (utf_to_sjis.map)
Another problem is vendor extension.
There are several standards for SJIS and EUC_JP in Japan. There is a
standard "Shift_JIS" defined by Japanese Government (probably the
Unicode consortium's map can be based on this, but I need to
verify). However several major vendors include IBM, NEC added their
own additional characters to Shift_JIS and they are widely used in
Japan. Unfortunately they are not compatible. So as a compromise I and
other developers decided to "merge" NEC and IBM extension part and
added to Shift_JIS. Same thing can be said to EUC_JP.
In short, there are number of reasons we cannot simply import the
consortium's mapping regarding SJIS (and EUC_JP).
Anyway, this seems to mean that it's okay to go ahead with fixing the
encoding conversion discrepancies I complained of yesterday; the data
those proposed diffs are based on is solid. But we've evidently got
a number of other issues with these Far Eastern encodings.regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
I wrote:
There's a discussion over at
/messages/by-id/2sa.Dhu5.1hk1yrpTNFy.1MLOlb@seznam.cz
of an apparent error in our WIN1250 -> LATIN2 conversion.
Attached is an updated patch (against today's HEAD) showing proposed
changes to bring cyrillic_and_mic.c and latin2_and_win1250.c into sync
with the Unicode Consortium's conversion data.
In addition, I've attached the C program I used to generate the proposed
new conversion tables from the Unicode/*.map files, a simple SQL script
to print out the conversion behavior for the affected conversions, and
a diff of the script's output between 9.5 and the proposed patch.
While the changes in the WIN1250 <-> LATIN2 conversions just amount to
removal of some translations that seem to have no basis in reality, the
changes in the Cyrillic mappings are quite a bit more extensive. It would
be good if we could get those checked by some native Russian speakers.
regards, tom lane
Attachments:
encoding-conversion-corrections-2.patchtext/x-diff; charset=us-ascii; name=encoding-conversion-corrections-2.patchDownload+116-114
buildmap.ctext/x-c; charset=us-ascii; name=buildmap.cDownload
checkconv.sqltext/plain; charset=us-ascii; name=checkconv.sqlDownload
diffs9.5vspatchtext/x-diff; charset=us-ascii; name=diffs9.5vspatchDownload+416-419
On Fri, Nov 27, 2015 at 8:54 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
I explain why the manual editing is necessary.
One of the most famous problems with Unicode is "wave dash"
(U+301C). According the Unicode consortium's Unicode/SJIS map, it
corresponds to 0x8160 of Shift_JIS. Unfortunately this was a mistake
in Unicode (the glyph of Shift_JIS and Unicode is slightly different -
looks like to be rotated in 90 degrees of wave dash in vertical
scripting. Probably they did not understand the Japanese vertical
writing at that time). So later on the Unicode consortium decided to
add another "wave dash" as U+FF5E which has a correct glyph of "wave
dash". However since Unicode already decided that U+301C corresponds
to 0x8160 of Shift_JIS, there's no Shift_JIS code corresponding to
U+FF5E. Unlike Unicode's definition, Microsoft defines that 0x8160
(wave dash) corresponds to U+FF5E. This is widely used in Japan. So I
decided to hire this for "wave dash". i.e.0x8160 -> U+FF5E (sjis_to_utf8.map)
U+301C -> 0x8160 (utf_to_sjis.map)
U+FF5E -> 0x8160 (utf_to_sjis.map)Another problem is vendor extension.
There are several standards for SJIS and EUC_JP in Japan. There is a
standard "Shift_JIS" defined by Japanese Government (probably the
Unicode consortium's map can be based on this, but I need to
verify). However several major vendors include IBM, NEC added their
own additional characters to Shift_JIS and they are widely used in
Japan. Unfortunately they are not compatible. So as a compromise I and
other developers decided to "merge" NEC and IBM extension part and
added to Shift_JIS. Same thing can be said to EUC_JP.In short, there are number of reasons we cannot simply import the
consortium's mapping regarding SJIS (and EUC_JP).
I haven't seen a response to this point, but it seems important.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Robert Haas <robertmhaas@gmail.com> writes:
On Fri, Nov 27, 2015 at 8:54 PM, Tatsuo Ishii <ishii@postgresql.org> wrote:
In short, there are number of reasons we cannot simply import the
consortium's mapping regarding SJIS (and EUC_JP).
I haven't seen a response to this point, but it seems important.
I'll defer to Tatsuo-san concerning whether the Far Eastern conversions
should act the way they do. However, I still think the Cyrillic and
Latin-2 conversions are broken. There is no reason to question the
Unicode consortium's mappings in those cases AFAIK, and even if somebody
wants to, our current tables fail to round-trip some characters, which
is surely wrong. (See the "inconsistent reverse conversion" complaints
in the test output in <32464.1448742262@sss.pgh.pa.us>.)
Regardless of that, it's dismaying that we have files in our tree that
claim to produce our mapping tables from authoritative sources, when in
fact those tables were not produced in that way. This is a documentation
failure even if you consider the actual conversion behavior valid.
regards, tom lane
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers