\w doesn't match non-ASCII letters

Started by Markus Bertheaualmost 22 years ago5 messagesbugs
Jump to latest
#1Markus Bertheau
twanger@bluetwanger.de

oocms=# select 'ф' ~ '^\\w$';
?column?
----------
f
(1 запись)

or

oocms=# select 'ä' ~ '^\\w$';
?column?
----------
f
(1 запись)

both should return true, as does

oocms=# select 'n' ~ '^\\w$';
?column?
----------
t
(1 запись)

Thanks.

--
Markus Bertheau <twanger@bluetwanger.de>

#2Peter Eisentraut
peter_e@gmx.net
In reply to: Markus Bertheau (#1)
Re: \w doesn't match non-ASCII letters

Markus Bertheau wrote:

oocms=# select 'ф' ~ '^\\w$';
?column?
----------
f
(1 запись)

What locale are you using for LC_COLLATE? If it's C or POSIX, you need
to change it and re-initdb.

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Peter Eisentraut (#2)
Re: \w doesn't match non-ASCII letters

Peter Eisentraut <peter_e@gmx.net> writes:

Markus Bertheau wrote:

oocms=# select 'ф' ~ '^\\w$';
?column?
----------
f
(1 запись)

What locale are you using for LC_COLLATE? If it's C or POSIX, you need
to change it and re-initdb.

Another likely cause of trouble is that the regexp character
classification stuff is presently based on <ctype.h> functions and thus
cannot work in multibyte encodings.

regards, tom lane

#4Markus Bertheau
twanger@bluetwanger.de
In reply to: Tom Lane (#3)
Re: \w doesn't match non-ASCII letters

В Пнд, 14.06.2004, в 17:25, Tom Lane пишет:

Peter Eisentraut <peter_e@gmx.net> writes:

Markus Bertheau wrote:

oocms=# select 'ф' ~ '^\\w$';
?column?
----------
f
(1 запись)

What locale are you using for LC_COLLATE? If it's C or POSIX, you need
to change it and re-initdb.

Another likely cause of trouble is that the regexp character
classification stuff is presently based on <ctype.h> functions and thus
cannot work in multibyte encodings.

This is in a UTF-8 database, so yes, these are multibyte characters. Is
there something planned to support UTF-8 in regexps?

--
Markus Bertheau <twanger@bluetwanger.de>

#5Tom Lane
tgl@sss.pgh.pa.us
In reply to: Markus Bertheau (#4)
Re: \w doesn't match non-ASCII letters

Markus Bertheau <twanger@bluetwanger.de> writes:

Is there something planned to support UTF-8 in regexps?

It'd be relatively easy to use the <wctype.h> functions here if we
were convinced that pg_mb2wchar() generated exactly the same
wide-character encoding as the C library is expecting for the current
LC_CTYPE setting. In the absence of such a guarantee I think we'd
have to convert the pg_wchar back to multibyte form and then apply
mbstowcs(), which is rather painful, not least because our wide
character support doesn't seem to have any function for converting
back to multibyte form ...

Tatsuo, any thoughts here?

regards, tom lane