client libpq multibyte support

SAKAIDA Masaaki <sakaida@psn.co.jp> writes:

A client application using libpq made by non-MULTIBYTE
can not talk to server made by MULTIBYTE.

admin=# select * from SJIS_KANJI ;
\: extra argument ';' ignored
\: extra argument ';' ignored
Invalid command \. Try \? for help.

Ugh :-(. We have not seen this reported before --- do you know exactly
where it's coming from? (I suspect it may be a psql issue not a libpq
issue, but hard to say without more info.)

I hope that a client 7.0-libpq and an application always be
made by "configure --enable-multibyte" even if MULTIBYTE isn't
necessary for backend. If so, the above problem will be solved.

I do not think that will go over well with people who don't need
multibyte support, since the MULTIBYTE code is a good deal larger
and slower. Also, AFAIK we didn't have any such problem in 6.5, so
perhaps this is just a small bug not requiring such a sledgehammer
solution. We need to look more closely.

regards, tom lane

admin=# select * from SJIS_KANJI ;
\: extra argument ';' ignored
\: extra argument ';' ignored
Invalid command \. Try \? for help.

Ugh :-(. We have not seen this reported before --- do you know exactly
where it's coming from? (I suspect it may be a psql issue not a libpq
issue, but hard to say without more info.)

That's because none-MB client does not understand how "Shift JIS
kanji" consists of letters with different width bytes. The similar
problem would happen with the Big5 character set (traditional
Chinese), also. Unlike other character sets, these should be treated
carefully since they include the same bit patterns as ASCII and that
makes none-MB clients confused.

I do not think that will go over well with people who don't need
multibyte support, since the MULTIBYTE code is a good deal larger
and slower. Also, AFAIK we didn't have any such problem in 6.5, so
perhaps this is just a small bug not requiring such a sledgehammer
solution. We need to look more closely.

No, 6.5 (and former versions) has exactly the same "bug." The reason
why you didn't hear it by now is that just nobody had tried to mixed
MB/none-MB backend/server configurations until Masaaki came up with
pgbash:-) Anyway, I could hardly imagine that such configurations
would actually exist in the real world. Masaaki, could you tell me
what are the advantages or reasons of the configuration?

For the Tom's comment of "the MULTIBYTE code is a good deal larger and
slower": IMHO it's a price of i18n (I don't claim my implementation of
MB is the most efficient one, though). Today almost any OS and
applications are evolving to be "i18n ready." Look at Lamar's new RPM.
The multibyte and the locale functionalities are now enabled by
default in it.

In the near future, PostgreSQL would have true i18n functionalities
(NATIONAL CHARACTER and friends), and I look forward to join the work.
I hope PostgreSQL would be i18n ready by default at that time.
--
Tatsuo Ishii

That's because none-MB client does not understand how "Shift JIS
kanji" consists of letters with different width bytes. The similar
problem would happen with the Big5 character set (traditional
Chinese), also. Unlike other character sets, these should be treated
carefully since they include the same bit patterns as ASCII and that
makes none-MB clients confused.

I'm confused though, this would mean that somewhere in the string
`SJIS_KANJI' a backslash was found. But that's all ASCII characters.
Aren't the characters 0-127 always identical in any character set?

Not always. Shift JIS and Big5 include 0-127 characters. So "how to
distinguish them from ASCII?", you might ask. Here are rules for this:

1. parse from the begining byte of the string in question. If it is
0-127 then it's an ASCII (single byte letter).

2. if it's between 0xa1 and 0xdf, it's a "1 byte kana" (single byte
letter).

3. otherwise it's a "kanji" (double byte letter). In this case the
second byte might be in range of 0-127 (this is the source of the
problem).

I think Big5 has similar, but a little bit different rule (I don't
remember precisely now).

Other encodings having 0-127 range bytes (but they are not ASCII)
include:

o UCS-2, 4 (Unicode)

o any 7 bit encoded ISO 2022 based charsets. for example, ISO 2022-jp.
--
Tatsuo Ishii

Tatsuo Ishii <t-ishii@sra.co.jp> writes:

For the Tom's comment of "the MULTIBYTE code is a good deal larger and
slower": IMHO it's a price of i18n (I don't claim my implementation of
MB is the most efficient one, though). Today almost any OS and
applications are evolving to be "i18n ready."

True, and in fact most of the performance problem in the client-side
MULTIBYTE code comes from the fact that it's not designed-in, but tries
to be a minimally intrusive patch. I think we could make it go faster
if we accepted that it was standard functionality. So I'm not averse to
going in that direction in the long term ... but I do object to turning
on MULTIBYTE by default just a couple days before release. We don't
really know how robust the MULTIBYTE-client-and-non-MULTIBYTE-server
combination is, and so I'm afraid to make it the default configuration
with hardly any testing.

regards, tom lane

True, and in fact most of the performance problem in the client-side
MULTIBYTE code comes from the fact that it's not designed-in, but tries
to be a minimally intrusive patch. I think we could make it go faster
if we accepted that it was standard functionality. So I'm not averse to
going in that direction in the long term ...

Glad to hear that.

but I do object to turning
on MULTIBYTE by default just a couple days before release. We don't
really know how robust the MULTIBYTE-client-and-non-MULTIBYTE-server
combination is, and so I'm afraid to make it the default configuration
with hardly any testing.

Agreed.
--
Tatsuo Ishii

Tatsuo Ishii <t-ishii@sra.co.jp> wrote:

admin=# select * from SJIS_KANJI ;
\: extra argument ';' ignored
\: extra argument ';' ignored
Invalid command \. Try \? for help.

(snip)

That's because none-MB client does not understand how "Shift JIS
kanji" consists of letters with different width bytes. The similar
problem would happen with the Big5 character set (traditional
Chinese), also. Unlike other character sets, these should be treated
carefully since they include the same bit patterns as ASCII and that
makes none-MB clients confused.

Thank you for your reply.

(Probably, the direct cause of this error is PQmblen(). non-
MULTIBYTE-PQmblen() always return "1". )

Anyway, I could hardly imagine that such configurations
would actually exist in the real world. Masaaki, could you tell me
what are the advantages or reasons of the configuration?

# My poor English won't be able to explain the real world ;-).

If a client libpq always be made by "configure --enable-
multibute", the advantages are

1. In the case of SQL_ASCII, a client application speed is
almost equal to non-MULTIBYTE. And the MULTIBYTE code is
not so larger.
2. When required, by using "set client_encoding=xxx", it is
possible to use the MULTIBYTE at any time.

--
Regard,
SAKAIDA Masaaki -- Osaka, Japan

True, and in fact most of the performance problem in the client-side
MULTIBYTE code comes from the fact that it's not designed-in, but tries
to be a minimally intrusive patch. I think we could make it go faster
if we accepted that it was standard functionality. So I'm not averse to
going in that direction in the long term ...

Glad to hear that.

but I do object to turning
on MULTIBYTE by default just a couple days before release. We don't
really know how robust the MULTIBYTE-client-and-non-MULTIBYTE-server
combination is, and so I'm afraid to make it the default configuration
with hardly any testing.

Agreed.

Thank you for your challenge. I expect that a good result comes out.

--
Regard,
SAKAIDA Masaaki -- Osaka, Japan

Please allow me to pick out this thread again.

True, and in fact most of the performance problem in the client-side
MULTIBYTE code comes from the fact that it's not designed-in, but tries
to be a minimally intrusive patch. I think we could make it go faster
if we accepted that it was standard functionality. So I'm not averse to
going in that direction in the long term ...

I have checked the performance problem.

(Environment)
- Hardware : P200pro CPU, 128MB, 5400rpm disk
- OS : Red hat Linux-5.2
- Database version : postgresql-7.0RC1

(Tested software and data)
- Library : libpq
- Program : ecpg application program, psql
- SQL : insert, select
- Number of tuples : 100,000 tuples

(Test case)
(1) non-MULTIBYTE
(2) MULTIBYTE encoding=SQL_ASCII

An ecpg program and the psql were used in this test case.

(Result)
As for the result, there was no difference in the speed of (1)
and (2). I could *not* find the performance problem.

(Improvement)
However, the performance problem may occur if the test of
10,000,000 tuples will be done. Because PQmblen() has a little
overhead of routine-call. Therefore, if the MULTIBYTE PQmblen()
will be changed as the following, the perfomance problem disappers
*perfectly*.

# ifdef MULTIBYTE
int PQmblen(const unsigned char *s, int encoding){
if( encoding == SQL_ASCII ) return 1; <======= Added line
return (pg_encoding_mblen(encoding, s));
}
# endif

(Conclusion)
A client library/application should be made by "configure
--enable-multibyte[=SQL_ASCII]" when postgresql is made by
"configure [non-MULTIBYTE]".

(Reference of library size)
non-MULTIBYTE MULTIBYTE
libpq.a 69KB 91KB
libpq.so.2.0 52KB 52KB
libpq.so.2.1 60KB 78KB

--
Regard,
SAKAIDA Masaaki -- Osaka, Japan