upper()/lower() truncates the result under Japanese Windows
Hi,
Upper(), lower() or initcap() function truncates the result
under Japanese Windows with e.g. the server encoding=UTF-8
and the LC_CTYPE setting Japanese_japan.932 .
Below is an example.
$ psql
psql (8.4devel)
Type "help" for help.
inoue=# \encoding sjis
inoue=# show server_encoding;
server_encoding
-----------------
UTF8
(1 行)
inoue=# show LC_CTYPE;
lc_ctype
--------------------
Japanese_Japan.932
(1 行)
inoue=# \set jpnstr '''カタカナ'''
inoue=# select char_length(:jpnstr);
char_length
-------------
4
(1 行)
inoue=# select upper(:jpnstr);
upper
--------
カタカ
(1 行)
inoue=# select char_length(upper(:jpnstr));
char_length
-------------
3
(1 行)
The output of the last command should be 4 not 3.
Attached is a patch to fix the bug.
After applying the patch the result is
inoue=# select upper(:jpnstr);
upper
----------
カタカナ
(1 行)
inoue=# select char_length(upper(:jpnstr));
char_length
-------------
4
(1 行)
regards,
Hiroshi Inoue
Attachments:
formatting.patchtext/plain; name=formatting.patchDownload
Index: formatting.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/formatting.c,v
retrieving revision 1.151
diff -c -c -r1.151 formatting.c
*** formatting.c 1 Dec 2008 17:11:18 -0000 1.151
--- formatting.c 14 Dec 2008 09:09:00 -0000
***************
*** 1462,1467 ****
--- 1462,1468 ----
{
wchar_t *workspace;
int curr_char = 0;
+ size_t max_len, alloc_size;
/* Output workspace cannot have more codes than input bytes */
workspace = (wchar_t *) palloc((nbytes + 1) * sizeof(wchar_t));
***************
*** 1472,1480 ****
workspace[curr_char] = towlower(workspace[curr_char]);
/* Make result large enough; case change might change number of bytes */
! result = palloc(curr_char * MB_CUR_MAX + 1);
! wchar2char(result, workspace, curr_char * MB_CUR_MAX + 1);
pfree(workspace);
}
else
--- 1473,1489 ----
workspace[curr_char] = towlower(workspace[curr_char]);
/* Make result large enough; case change might change number of bytes */
! #ifdef WIN32
! max_len = pg_database_encoding_max_length();
! if (MB_CUR_MAX > max_len)
! max_len = MB_CUR_MAX;
! #else
! max_len = MB_CUR_MAX;
! #endif
! alloc_size = curr_char * max_len + 1;
! result = palloc(alloc_size);
! wchar2char(result, workspace, alloc_size);
pfree(workspace);
}
else
***************
*** 1510,1515 ****
--- 1519,1525 ----
{
wchar_t *workspace;
int curr_char = 0;
+ size_t max_len, alloc_size;
/* Output workspace cannot have more codes than input bytes */
workspace = (wchar_t *) palloc((nbytes + 1) * sizeof(wchar_t));
***************
*** 1520,1528 ****
workspace[curr_char] = towupper(workspace[curr_char]);
/* Make result large enough; case change might change number of bytes */
! result = palloc(curr_char * MB_CUR_MAX + 1);
! wchar2char(result, workspace, curr_char * MB_CUR_MAX + 1);
pfree(workspace);
}
else
--- 1530,1546 ----
workspace[curr_char] = towupper(workspace[curr_char]);
/* Make result large enough; case change might change number of bytes */
! #ifdef WIN32
! max_len = pg_database_encoding_max_length();
! if (MB_CUR_MAX > max_len)
! max_len = MB_CUR_MAX;
! #else
! max_len = MB_CUR_MAX;
! #endif
! alloc_size = curr_char * max_len + 1;
! result = palloc(alloc_size);
! wchar2char(result, workspace, alloc_size);
pfree(workspace);
}
else
***************
*** 1559,1564 ****
--- 1577,1583 ----
{
wchar_t *workspace;
int curr_char = 0;
+ size_t max_len, alloc_size;
/* Output workspace cannot have more codes than input bytes */
workspace = (wchar_t *) palloc((nbytes + 1) * sizeof(wchar_t));
***************
*** 1575,1583 ****
}
/* Make result large enough; case change might change number of bytes */
! result = palloc(curr_char * MB_CUR_MAX + 1);
! wchar2char(result, workspace, curr_char * MB_CUR_MAX + 1);
pfree(workspace);
}
else
--- 1594,1610 ----
}
/* Make result large enough; case change might change number of bytes */
! #ifdef WIN32
! max_len = pg_database_encoding_max_length();
! if (MB_CUR_MAX > max_len)
! max_len = MB_CUR_MAX;
! #else
! max_len = MB_CUR_MAX;
! #endif
! alloc_size = curr_char * max_len + 1;
! result = palloc(alloc_size);
! wchar2char(result, workspace, alloc_size);
pfree(workspace);
}
else
Hiroshi Inoue <inoue@tpf.co.jp> writes:
Upper(), lower() or initcap() function truncates the result
under Japanese Windows with e.g. the server encoding=UTF-8
and the LC_CTYPE setting Japanese_japan.932 .
Hmm, I guess that makes sense, since the LC_CTYPE implies an encoding
other than UTF-8; MB_CUR_MAX should be set according to LC_CTYPE.
The proposed patch seems pretty ugly though. Why don't we just stop
using MB_CUR_MAX altogether? These three functions are the only
references to it AFAICS.
regards, tom lane
Tom Lane wrote:
Hiroshi Inoue <inoue@tpf.co.jp> writes:
Upper(), lower() or initcap() function truncates the result
under Japanese Windows with e.g. the server encoding=UTF-8
and the LC_CTYPE setting Japanese_japan.932 .Hmm, I guess that makes sense, since the LC_CTYPE implies an encoding
other than UTF-8; MB_CUR_MAX should be set according to LC_CTYPE.The proposed patch seems pretty ugly though. Why don't we just stop
using MB_CUR_MAX altogether? These three functions are the only
references to it AFAICS.
Although it looks ugly, it only follows what wchar2char() does.
Though I don't like to use MB_CUR_MAX, it seems safe as long as
wchar2char() calls wcstombs().
regards,
Hiroshi Inoue