BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters

=?utf-8?q?PG_Bug_reporting_form?= <noreply@postgresql.org> writes:

On Encoding=UTF-8 database, try:
SELECT show_trgm('123');
→ OK
SELECT show_trgm('日本語');
→ probably OK.
SELECT show_trgm('🔍');
ERROR: invalid multibyte character for locale
HINT: The server's LC_CTYPE locale is probably incompatible with the
database encoding.
SQL state: 22021

I failed to reproduce this on a Linux machine. It looks to me like the
problem is that Windows' MultiByteToWideChar doesn't think that UTF8
character is valid.

Please check: t_isdigit, t_isspace, t_isalpha, and t_isprint.
https://github.com/postgres/postgres/blob/322548a8abe225f2cfd6a48e07b99e2711d28ef7/src/backend/tsearch/ts_locale.c#L35
char2wchar 4th parameter should take number of input bytes. However they
pass character count.
int clen = pg_mblen(ptr);
...
char2wchar(character, 2, ptr, clen, mylocale);

Huh? pg_mblen returns the number of bytes in a multibyte character,
so this looks fine to me.

regards, tom lane

kenji uno <h8mastre@gmail.com> writes:

I failed to reproduce this on a Linux machine. It looks to me like the
problem is that Windows' MultiByteToWideChar doesn't think that UTF8
character is valid.

I'm just wondering why my issue occurs only on Windows.
But I knew why: char2wchar's tolen requires +1 output buffer size, due to
null-termination.

Oooh ... the problem, effectively, is that the ts_locale.c functions are
expecting to get back UTF32 but what they'll actually get on Windows is
UTF16. So if the given character is outside the BMP range, char2wchar
needs to produce a surrogate pair, which there's not room for given that
the output buffer can only hold 1 wchar_t plus trailing null.

Then the other problem is that the Windows-Unicode code path in char2wchar
just fails for an undersized output buffer, which you would not expect
from its documentation. And it fails with a misleading error message,
too.

I'll see what I can do about this --- thanks for the report!

regards, tom lane