BUG #1931: ILIKE and LIKE fails on Turkish locale

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

http://sourceware.org/bugzilla/long_list.cgi?buglist=1354
So it is PostgreSQL's bug or Glibc's?

Just offhand, iwchareq() seems several bricks shy of a load:

/*
* if one of them is an ASCII while the other is not, then they must
* be different characters
*/
else if ((unsigned char) *p1 < CHARMAX || (unsigned char) *p2 < CHARMAX)
return (0);

This test is wrong per Jakub's observation. Also, the code right below
that is using tolower() not towlower() on wide characters, which seems
pretty wrong. For that matter, towlower would be wrong too :-( because
there is no certainty that libc's idea of wide characters is the same as
pg_mb2wchar_with_len's.

So yeah, ILIKE looks just about completely broken for multibyte encodings.
Maybe it would be best to pass both strings through lower() and then
do a normal LIKE comparison?

The regexp code doesn't look better, btw, just differently broken ...

regards, tom lane

Hi Tom,

On Sat, 1 Oct 2005, Tom Lane wrote:

<snip>

So yeah, ILIKE looks just about completely broken for multibyte encodings.
Maybe it would be best to pass both strings through lower() and then
do a normal LIKE comparison?

The regexp code doesn't look better, btw, just differently broken ...

So any chance to fix it before beta 3 (and also backpatch?) ?

Regards,
--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-bugs-owner@postgresql.org Sun Oct 2 00:14:30 2005

X-Original-To: pgsql-bugs-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id 0E04ED8BF7;
Sun, 2 Oct 2005 00:14:21 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 37446-07; Sun, 2 Oct 2005 03:14:19 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
by svr1.postgresql.org (Postfix) with ESMTP id 59D05D889A;
Sun, 2 Oct 2005 00:14:18 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j923EDNq029724;
Sat, 1 Oct 2005 23:14:15 -0400 (EDT)
To: Devrim GUNDUZ <devrim@gunduz.org>
Cc: pgsql-bugs@postgresql.org, pgsql-tr-genel@postgresql.org
Subject: Re: BUG #1931: ILIKE and LIKE fails on Turkish locale
In-reply-to: <Pine.LNX.4.63.0510020336270.25019@mail.kivi.com.tr>
References: <20051001153536.6C9EBF0B6F@svr2.postgresql.org> <9028.1128184301@sss.pgh.pa.us> <Pine.LNX.4.63.0510020336270.25019@mail.kivi.com.tr>
Comments: In-reply-to Devrim GUNDUZ <devrim@gunduz.org>
message dated "Sun, 02 Oct 2005 03:38:01 +0300"
Date: Sat, 01 Oct 2005 23:14:13 -0400
Message-ID: <29723.1128222853@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Spam-Level:
X-Archive-Number: 200510/5
X-Sequence-Number: 12986

Devrim GUNDUZ <devrim@gunduz.org> writes:

On Sat, 1 Oct 2005, Tom Lane wrote:

So yeah, ILIKE looks just about completely broken for multibyte encodings.
The regexp code doesn't look better, btw, just differently broken ...

So any chance to fix it before beta 3 (and also backpatch?) ?

Are you volunteering?

This stuff has never worked, so I don't feel that it qualifies as "must
fix for 8.1". If you can come up with a believable low-impact patch,
I'd support putting it in now ... but my guess is that this is a
significant bit of work that will not get done in this release cycle.

regards, tom lane

:)))))))))))))))))))))))))))

Selamlar

12 g�nd�r bu problem �zerinde �al���yoruz. Projem i�in Zeoslib
kullan�yordum. Sorun onda diye zeos u alt�n� �st�ne getirdim. Araya ba�ka
i�ler girdi ve ara verdim. Siz yaz�nca benim tekrar akl�ma geldi. Yaf
PostgreSQL in b�yle basit bir hataya d��ebilece�ini hi� d���nmemi�tim :(
Neyseki siz buldunuz da telef olmaktan kurtuldum.

�yi �al��malar..

Ali �EL�K

"Devrim GUNDUZ" <devrim@gunduz.org>, haber iletisinde �unlar�
yazd�:Pine.LNX.4.63.0510020336270.25019@mail.kivi.com.tr...

Merhaba,

On Sun, 2 Oct 2005, AL� �EL�K wrote:

12 g�nd�r bu problem �zerinde �al���yoruz. Projem i�in Zeoslib
kullan�yordum. Sorun onda diye zeos u alt�n� �st�ne getirdim. Araya ba�ka
i�ler girdi ve ara verdim. Siz yaz�nca benim tekrar akl�ma geldi. Yaf
PostgreSQL in b�yle basit bir hataya d��ebilece�ini hi� d���nmemi�tim :(
Neyseki siz buldunuz da telef olmaktan kurtuldum.

Nicolai ile bunun �zerinde �al��aca��z ama san�r�m 8.1'e yeti�meyecek.
8.1.X'e yeti�mesini �mit ediyoruz (ger�i 8.2 i�in de olabilir ama o ge�
olur).

�lgi bilgi.
--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-tr-genel-owner@postgresql.org Tue Oct 4 18:51:23 2005

X-Original-To: pgsql-tr-genel-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id 1B630DA0B3;
Tue, 4 Oct 2005 18:51:20 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 90430-08; Tue, 4 Oct 2005 21:51:14 +0000 (GMT)
Received: from mail.kivi.com.tr (mail.kivi.com.tr [213.143.254.37])
by svr1.postgresql.org (Postfix) with ESMTP id 76C67DA0A9;
Tue, 4 Oct 2005 18:51:15 -0300 (ADT)
Received: by mail.kivi.com.tr (Postfix, from userid 1009)
id 4F366E81D6; Wed, 5 Oct 2005 00:50:49 +0300 (EEST)
Received: from localhost (localhost [127.0.0.1])
by mail.kivi.com.tr (Postfix) with ESMTP id 453D3E81CB;
Wed, 5 Oct 2005 00:50:49 +0300 (EEST)
Date: Wed, 5 Oct 2005 00:50:49 +0300 (EEST)
From: Devrim GUNDUZ <devrim@gunduz.org>
X-X-Sender: devrim2@mail.kivi.com.tr
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-bugs@postgresql.org, pgsql-tr-genel@postgresql.org
Subject: Re: [BUGS] BUG #1931: ILIKE and LIKE fails on Turkish locale
In-Reply-To: <29723.1128222853@sss.pgh.pa.us>
Message-ID: <Pine.LNX.4.63.0510021152490.25019@mail.kivi.com.tr>
References: <20051001153536.6C9EBF0B6F@svr2.postgresql.org>
<9028.1128184301@sss.pgh.pa.us> <Pine.LNX.4.63.0510020336270.25019@mail.kivi.com.tr>
<29723.1128222853@sss.pgh.pa.us>
MIME-Version: 1.0
X-Kivi-MailScanner-Information: Please contact the ISP for more information
X-Kivi-MailScanner: Found to be clean
X-Kivi-MailScanner-From: devrim@gunduz.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Spam-Level:
Content-Type: TEXT/PLAIN; CHARSET=iso-8859-9; FORMAT=flowed
Content-Transfer-Encoding: 8BIT
Content-ID: <Pine.LNX.4.63.0510050050171.15246@mail.kivi.com.tr>
X-Archive-Number: 2005101/9
X-Sequence-Number: 299

Hi,

On Sat, 1 Oct 2005, Tom Lane wrote:

So yeah, ILIKE looks just about completely broken for multibyte encodings.
The regexp code doesn't look better, btw, just differently broken ...

So any chance to fix it before beta 3 (and also backpatch?) ?

Are you volunteering?

This stuff has never worked, so I don't feel that it qualifies as "must
fix for 8.1". If you can come up with a believable low-impact patch,
I'd support putting it in now ... but my guess is that this is a
significant bit of work that will not get done in this release cycle.

Yeah, I and Nicolai Tufar volunteer to fix it.

Regards,
--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-tr-genel-owner@postgresql.org Tue Oct 4 19:00:35 2005

X-Original-To: pgsql-tr-genel-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id 2737BDA09D
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>; Tue, 4 Oct 2005 19:00:33 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 97932-03
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>;
Tue, 4 Oct 2005 22:00:29 +0000 (GMT)
Received: from mail.kivi.com.tr (mail.kivi.com.tr [213.143.254.37])
by svr1.postgresql.org (Postfix) with ESMTP id 078FADA07E
for <pgsql-tr-genel@postgresql.org>; Tue, 4 Oct 2005 19:00:29 -0300 (ADT)
Received: by mail.kivi.com.tr (Postfix, from userid 1009)
id F2998E81D6; Wed, 5 Oct 2005 01:00:02 +0300 (EEST)
Received: from localhost (localhost [127.0.0.1])
by mail.kivi.com.tr (Postfix) with ESMTP id E906EE81CB;
Wed, 5 Oct 2005 01:00:02 +0300 (EEST)
Date: Wed, 5 Oct 2005 01:00:02 +0300 (EEST)
From: Devrim GUNDUZ <devrim@gunduz.org>
X-X-Sender: devrim2@mail.kivi.com.tr
Reply-To: pgsql-tr-genel@postgresql.org
To: =?iso-8859-9?Q?AL=DD_=C7EL=DDK?= <ali@verus.com.tr>
Cc: pgsql-tr-genel@postgresql.org
Subject: Re: FreeBSD vs Linux
In-Reply-To: <dhoq15$1jat$1@news.hub.org>
Message-ID: <Pine.LNX.4.63.0510021739200.25019@mail.kivi.com.tr>
References: <dhoq15$1jat$1@news.hub.org>
MIME-Version: 1.0
X-Kivi-MailScanner-Information: Please contact the ISP for more information
X-Kivi-MailScanner: Found to be clean
X-Kivi-MailScanner-From: devrim@gunduz.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Spam-Level:
Content-Type: TEXT/PLAIN; CHARSET=iso-8859-9; FORMAT=flowed
Content-Transfer-Encoding: 8BIT
Content-ID: <Pine.LNX.4.63.0510050052591.15246@mail.kivi.com.tr>
X-Archive-Number: 2005101/10
X-Sequence-Number: 300

Merhaba,

On Sun, 2 Oct 2005, AL� �EL�K wrote:

PostgreSQL in Hangi platformda ne kadar ba�ar�m verdi�i ile ilgili bir
istatistik var m�?

Linux,FreeBSD veya Win32 performanslar� nas�l?

Win32 s�r�m� hen�z yeni say�l�r. Her ne kadar i�indeki b�y�k kod par�as�
eski olsa da Windows �zerindeki kararl�l�k i�in bence biraz daha beklemek
gerekli. (Belki performans ile ilgili de�il ama) Unicode vs deste�i de 8.0
i�inde yok (Windows s�r�m�nde), bu nedenle tercih edilmez diye
d���nmekteyim.

FreeBSD ve Linux'a gelince...

�imdi benim biri hari� t�m sunucular�m Linux. FreeBSD olana yeni kurdum ve
y�netmeye ba�lad�m, o y�zden net bir kar��la�t�rma yapamayaca��m.

Ancak, PostgreSQL.org sunucular�n�n y�netiminden edindi�im tecr�beye
bakarsak, ikisi aras�nda birbirini " ezecek " kadar bir fark yok.

Ben RHEL ve SLES �zerinde deniyorum son zamanlarda. RHEL 3.0 �zerinde
ciddi bir sunucu �al���yor:

[root@web pg_log]# ll
total 1367172
-rw------- 1 postgres postgres 184232379 Oct 1 00:00
postgresql-Fri.log
-rw------- 1 postgres postgres 334944335 Oct 4 00:00
postgresql-Mon.log
-rw------- 1 postgres postgres 193482214 Oct 1 23:59
postgresql-Sat.log
-rw------- 1 postgres postgres 164792772 Oct 2 23:59
postgresql-Sun.log
-rw------- 1 postgres postgres 376297934 Sep 29 23:59
postgresql-Thu.log
-rw------- 1 postgres postgres 143517880 Oct 5 00:00
postgresql-Tue.log
-rw------- 1 postgres postgres 1300505 Oct 5 00:57 postgresql-Wed.log

Sunucunun �retti�i "g�nl�k" kay�tlar�n b�y�kl���ne bakarsan�z yo�unlu�u
tahmin edebilirsiniz. Bu sunucu 1 y�ldan beri �al���yor kesintisiz ve
gayet iyi bir durumda.

SLES �zerinde teknik testler kadar UNICODE testleri de ba�ar�l�. %100
bitiremedim testleri ama SLES de iyi bir alternatif olabilir.

Bu arada OSDL'de yap�lan testlere de bir bak�n. Orada 16-64 i�lemcili
makinelerde PostgreSQL deniyorlar.

adresine yollayabilirsiniz.

Her iletinin alt�na ekleniyor bu, nasil oldu�unu bulursam mutlu olaca��m
:(

--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-tr-genel-owner@postgresql.org Wed Oct 5 02:19:30 2005

X-Original-To: pgsql-tr-genel-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id CF7B8DA197
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>; Wed, 5 Oct 2005 02:17:54 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 68071-09
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>;
Wed, 5 Oct 2005 05:17:52 +0000 (GMT)
Received: from mail.kivi.com.tr (mail.kivi.com.tr [213.143.254.37])
by svr1.postgresql.org (Postfix) with ESMTP id 39659DA272
for <pgsql-tr-genel@postgresql.org>; Wed, 5 Oct 2005 02:17:50 -0300 (ADT)
Received: by mail.kivi.com.tr (Postfix, from userid 1009)
id BAD3CE81D6; Wed, 5 Oct 2005 08:17:20 +0300 (EEST)
Received: from localhost (localhost [127.0.0.1])
by mail.kivi.com.tr (Postfix) with ESMTP id AE689E81CB;
Wed, 5 Oct 2005 08:17:20 +0300 (EEST)
Date: Wed, 5 Oct 2005 08:17:20 +0300 (EEST)
From: Devrim GUNDUZ <devrim@gunduz.org>
X-X-Sender: devrim2@mail.kivi.com.tr
To: pgsql-tr-genel@PostgreSQL.org
Cc: linux-programlama@liste.linux.org.tr
Subject: =?iso-8859-9?Q?Yeni_PostgreSQL_ara_s=FCr=FCmleri_duyuruldu=2E?=
Message-ID: <Pine.LNX.4.63.0510050816410.20542@mail.kivi.com.tr>
MIME-Version: 1.0
X-Kivi-MailScanner-Information: Please contact the ISP for more information
X-Kivi-MailScanner: Found to be clean
X-Kivi-MailScanner-From: devrim@gunduz.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Spam-Level:
Content-Type: TEXT/PLAIN; charset=iso-8859-9; format=flowed
Content-Transfer-Encoding: 8BIT
X-Archive-Number: 2005101/11
X-Sequence-Number: 301

Merhaba,

Duyuru a�a��da:

---------- Forwarded message ----------
Date: Tue, 4 Oct 2005 20:15:16 -0300 (ADT)
From: Marc G. Fournier <scrappy@postgresql.org>
To: pgsql-announce@postgresql.org
Cc: pgsql-general@postgresql.org
Subject: [ANNOUNCE] New Point Releases Now Available

In order to address several issues identified since our last Point Releases,
we have just released the following new versions of PostgreSQL: 7.3.11, 7.4.9
and 8.0.4.

A few of the more prominent fixes across all three versions are:

- Fix error that allowed "VACUUM" to remove ctid chains too soon
- Fix missing rows in queries like UPDATE a=... WHERE a... with GiST
index on column a
- Improve checking for partially-written WAL pages
- Various memory leakage fixes

For a complete list of fixes in each release, please see the HISTORY file.

Source downloads are available at the following URLs:

http://www.postgresql.org/ftp/source/v7.3.11

http://www.postgresql.org/ftp/source/v7.4.9

http://www.postgresql.org/ftp/source/v8.0.4

Linux/Windows Binaries for 8.0.4 are available via:

http://www.postgresql.org/ftp/binary/v8.0.4

These Point Releases do not require a dump/reload from the previous Point,
but *may* require one if upgrading from an earlier version. Please see the
Release Notes to confirm procedures for upgrading if your current version is
older then 1 Point Release.

Marc G. Fournier
PostgreSQL Core Team

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match

From pgsql-tr-genel-owner@postgresql.org Wed Oct 5 17:23:28 2005

X-Original-To: pgsql-tr-genel-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id E6910DA26F
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>; Wed, 5 Oct 2005 17:23:25 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 47443-05
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>;
Wed, 5 Oct 2005 20:23:18 +0000 (GMT)
Received: from mail.kivi.com.tr (mail.kivi.com.tr [213.143.254.37])
by svr1.postgresql.org (Postfix) with ESMTP id EFB1BDA215
for <pgsql-tr-genel@postgresql.org>; Wed, 5 Oct 2005 17:23:15 -0300 (ADT)
Received: by mail.kivi.com.tr (Postfix, from userid 1009)
id 3DF3CE81D6; Wed, 5 Oct 2005 23:22:46 +0300 (EEST)
Received: from localhost (localhost [127.0.0.1])
by mail.kivi.com.tr (Postfix) with ESMTP id 3059AE81CB;
Wed, 5 Oct 2005 23:22:46 +0300 (EEST)
Date: Wed, 5 Oct 2005 23:22:46 +0300 (EEST)
From: Devrim GUNDUZ <devrim@gunduz.org>
X-X-Sender: devrim2@mail.kivi.com.tr
Reply-To: pgsql-tr-genel@PostgreSQL.org
To: pgsql-tr-genel@PostgreSQL.org
Cc: linux-programlama@liste.linux.org.tr
Subject: =?iso-8859-9?Q?Sun_-_PostgreSQL_yak=FDnla=FEmas=FD?=
Message-ID: <Pine.LNX.4.63.0510052317430.373@mail.kivi.com.tr>
MIME-Version: 1.0
X-Kivi-MailScanner-Information: Please contact the ISP for more information
X-Kivi-MailScanner: Found to be clean
X-Kivi-MailScanner-From: devrim@gunduz.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Spam-Level:
Content-Type: TEXT/PLAIN; CHARSET=iso-8859-9; FORMAT=flowed
Content-Transfer-Encoding: 8BIT
Content-ID: <Pine.LNX.4.63.0510052320301.373@mail.kivi.com.tr>
X-Archive-Number: 2005101/12
X-Sequence-Number: 302

Merhaba,

http://people.planetpostgresql.org/xzilla/index.php?/archives/91-Is-Sun-gettin
g-ready-to-join-the-PostgreSQL-community.html

http://www.computerworld.com.au/index.php/id;116679278;fp;16;fpid;0

http://developers.slashdot.org/developers/05/10/05/1344254.shtml?tid=221&amp;tid=218

Sayg�lar.
--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-tr-genel-owner@postgresql.org Thu Oct 6 13:13:00 2005

X-Original-To: pgsql-tr-genel-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id F3F8BDA1C4
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>; Thu, 6 Oct 2005 13:12:56 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 78986-07
for <pgsql-tr-genel-postgresql.org@localhost.postgresql.org>;
Thu, 6 Oct 2005 16:12:54 +0000 (GMT)
Received: from floppy.pyrenet.fr (floppy.pyrenet.fr [194.116.145.2])
by svr1.postgresql.org (Postfix) with ESMTP id 37A7ADA108
for <pgsql-tr-genel@postgresql.org>; Thu, 6 Oct 2005 13:12:49 -0300 (ADT)
Received: by floppy.pyrenet.fr (Postfix, from userid 106)
id 9ED9A31059; Thu, 6 Oct 2005 18:12:48 +0200 (MET DST)
From: "AL� �EL�K" <ali@verus.com.tr>
X-Newsgroups: pgsql.tr.genel
Subject: Tek sorguyla birden fazla veritaban�ndan bilgi �ekmek
Date: Thu, 6 Oct 2005 19:12:43 +0300
Organization: Hub.Org Networking Services
Lines: 7
Message-ID: <di3idt$kf5$1@news.hub.org>
X-Complaints-To: usenet@news.hub.org
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.2670
X-RFC2646: Format=Flowed; Original
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670
To: pgsql-tr-genel@postgresql.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char DD hex) in message header 'From': From: "AL\335 \307EL\335K" <ali@ve...
X-Spam-Status: No, hits=3.143 required=5 tests=[AWL=-0.896,
PRIORITY_NO_NAME=1.185, SUBJ_ILLEGAL_CHARS=2.854]
X-Spam-Level: ***
X-Archive-Number: 2005101/13
X-Sequence-Number: 303

Olabilir mi?

Hay�rl� ramazanlar...

Ali �EL�K

Hi,

On Sat, 1 Oct 2005, Tom Lane wrote:

So yeah, ILIKE looks just about completely broken for multibyte encodings.
The regexp code doesn't look better, btw, just differently broken ...

So any chance to fix it before beta 3 (and also backpatch?) ?

Are you volunteering?

This stuff has never worked, so I don't feel that it qualifies as "must
fix for 8.1". If you can come up with a believable low-impact patch,
I'd support putting it in now ... but my guess is that this is a
significant bit of work that will not get done in this release cycle.

Yeah, I and Nicolai Tufar volunteer to fix it.

Regards,
--
Devrim GUNDUZ
Kivi Bili�im Teknolojileri - http://www.kivi.com.tr
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

From pgsql-bugs-owner@postgresql.org Wed Oct 5 01:03:14 2005

X-Original-To: pgsql-bugs-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
by svr1.postgresql.org (Postfix) with ESMTP id A133FD8ED3
for <pgsql-bugs-postgresql.org@localhost.postgresql.org>; Wed, 5 Oct 2005 01:03:13 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
with ESMTP id 06112-04
for <pgsql-bugs-postgresql.org@localhost.postgresql.org>;
Wed, 5 Oct 2005 04:03:11 +0000 (GMT)
Received: from floppy.pyrenet.fr (news.pyrenet.fr [194.116.145.2])
by svr1.postgresql.org (Postfix) with ESMTP id 6452CD7FE7
for <pgsql-bugs@postgresql.org>; Wed, 5 Oct 2005 01:03:11 -0300 (ADT)
Received: by floppy.pyrenet.fr (Postfix, from userid 106)
id CD3F331059; Wed, 5 Oct 2005 06:03:10 +0200 (MET DST)
From: "Qingqing Zhou" <zhouqq@cs.toronto.edu>
X-Newsgroups: pgsql.bugs
Subject: Re: Possibly corrupted shared memory, PostgreSQL 8.1 beta2, Windows 2000
Date: Wed, 5 Oct 2005 00:03:10 -0700
Organization: Hub.Org Networking Services
Lines: 31
Message-ID: <dhvj9t$hel$1@news.hub.org>
References: <00b401c5c900$13abb410$6401a8c0@JP>
X-Complaints-To: usenet@news.hub.org
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2900.2180
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180
X-RFC2646: Format=Flowed; Response
To: pgsql-bugs@postgresql.org
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=1.19 required=5 tests=[AWL=0.005,
PRIORITY_NO_NAME=1.185]
X-Spam-Level: *
X-Archive-Number: 200510/43
X-Sequence-Number: 13024

""Jean-Pierre Pelletier"" <pelletier_32@sympatico.ca> wrote

I've installed PostgreSQL 8.1 beta2 five days ago and it crashed 3 times
since then.
Here is what's been logged for the last crash

2005-10-04 11:00:19 FATAL: could not read block 121 of relation
1663/16384/2608: Invalid argument

relation 2608 is pg_depend
----------------------------------------------------------------------------------
The crash before that was on relation pg_type, the first line logged was:
2005-10-03 10:51:06 FATAL: could not read block 38 of relation
1663/16384/1247: Invalid argument
----------------------------------------------------------------------------------
The first crash was also on relation pg_depend, but with open instead or
read
2005-09-30 18:38:53 FATAL: could not open relation 1663/16384/2608:
Invalid argument
----------------------------------------------------------------------------------

This problem was reported several times before, but not necessarily system
tables. Is there any anti-virus softwares installed on the same machine? Is
the database under intensive IO pressure?

Regards,
Qingqing

FreeBSD 5.4 de PostgreSQL 8.0.1 datas�n�

-E UNICODE --locale=tr_TR.UTF-8

parametresi ile kurdum. Sadece � -> i ve I -> � ili�kileri bozuk. San�r�m
sebebi ingilizce. :)
Harflerin boyu de�i�ince ki�ili�ide de�i�iyor.

Fakat LATIN5 (ISO 8859-9) ile yap�nca, hata t�m T�rk�e harfler de oluyordu.

Herkese iyi �al��malar.

"Devrim GUNDUZ" <devrim@gunduz.org>, haber iletisinde �unlar�
yazd�:Pine.LNX.4.63.0510021733050.25019@mail.kivi.com.tr...

This has been saved for the 8.2 release:

http://momjian.postgresql.org/cgi-bin/pgpatches_hold

---------------------------------------------------------------------------

Tom Lane wrote:

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

http://sourceware.org/bugzilla/long_list.cgi?buglist=1354
So it is PostgreSQL's bug or Glibc's?

Just offhand, iwchareq() seems several bricks shy of a load:

/*
* if one of them is an ASCII while the other is not, then they must
* be different characters
*/
else if ((unsigned char) *p1 < CHARMAX || (unsigned char) *p2 < CHARMAX)
return (0);

This test is wrong per Jakub's observation. Also, the code right below
that is using tolower() not towlower() on wide characters, which seems
pretty wrong. For that matter, towlower would be wrong too :-( because
there is no certainty that libc's idea of wide characters is the same as
pg_mb2wchar_with_len's.

So yeah, ILIKE looks just about completely broken for multibyte encodings.
Maybe it would be best to pass both strings through lower() and then
do a normal LIKE comparison?

The regexp code doesn't look better, btw, just differently broken ...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Did we make any progress on this? If so, I can't find it.

---------------------------------------------------------------------------

Tom Lane wrote:

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

http://sourceware.org/bugzilla/long_list.cgi?buglist=1354
So it is PostgreSQL's bug or Glibc's?

Just offhand, iwchareq() seems several bricks shy of a load:

/*
* if one of them is an ASCII while the other is not, then they must
* be different characters
*/
else if ((unsigned char) *p1 < CHARMAX || (unsigned char) *p2 < CHARMAX)
return (0);

This test is wrong per Jakub's observation. Also, the code right below
that is using tolower() not towlower() on wide characters, which seems
pretty wrong. For that matter, towlower would be wrong too :-( because
there is no certainty that libc's idea of wide characters is the same as
pg_mb2wchar_with_len's.

So yeah, ILIKE looks just about completely broken for multibyte encodings.
Maybe it would be best to pass both strings through lower() and then
do a normal LIKE comparison?

The regexp code doesn't look better, btw, just differently broken ...

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

So yeah, ILIKE looks just about completely broken for multibyte encodings.

Did we make any progress on this? If so, I can't find it.

No, nobody submitted a patch AFAIR. There should be a TODO item for
this:

* fix ILIKE and regular expressions to handle case insensitivity
properly in multibyte encodings

BTW, while looking to see if TODO had anything about this, I noted
the following items that shouldn't be there anymore --- they are
done and pushed out:

o Add new version of PQescapeString() that doesn't double backslashes
that are part of a client-only multibyte sequence

Single-quote is not a valid byte in any supported client-only
encoding. This requires using mblen() to determine if the
backslash is inside or outside a multi-byte sequence.

o Add new version of PQescapeString() that doesn't double
backslashes when standard_conforming_strings is true and
non-E strings are used

regards, tom lane

Thanks, both done.

---------------------------------------------------------------------------

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

Tom Lane wrote:

So yeah, ILIKE looks just about completely broken for multibyte encodings.

Did we make any progress on this? If so, I can't find it.

No, nobody submitted a patch AFAIR. There should be a TODO item for
this:

* fix ILIKE and regular expressions to handle case insensitivity
properly in multibyte encodings

BTW, while looking to see if TODO had anything about this, I noted
the following items that shouldn't be there anymore --- they are
done and pushed out:

o Add new version of PQescapeString() that doesn't double backslashes
that are part of a client-only multibyte sequence

Single-quote is not a valid byte in any supported client-only
encoding. This requires using mblen() to determine if the
backslash is inside or outside a multi-byte sequence.

o Add new version of PQescapeString() that doesn't double
backslashes when standard_conforming_strings is true and
non-E strings are used

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

--
Bruce Momjian http://candle.pha.pa.us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Jun 14 05:00, Bruce Momjian wrote:

Did we make any progress on this? If so, I can't find it.

I've made some tests for upper(), lower(), ILIKE and ~* using cvs tip.
Below are the details:

Cluster Locale | client_encoding | upper() | lower() | ILIKE | ~*
-----------------+-----------------+---------+---------+-------+----
tr_TR.iso8859-9 | LATIN5 | OK | OK | OK | OK
tr_TR.iso8859-9 | UTF8 | OK | OK | OK | OK
tr_TR.UTF-8 | LATIN5 | OK | OK | FAILS | FAILS
tr_TR.UTF-8 | UTF8 | OK | OK | FAILS | FAILS

Also, if you'd wish, I can prepare an ad-hoc regression tests patch
for LATIN5 and UTF-8 support of Turkish characters.

Regards.

Volkan YAZICI <yazicivo@ttnet.net.tr> writes:

Also, if you'd wish, I can prepare an ad-hoc regression tests patch
for LATIN5 and UTF-8 support of Turkish characters.

We know it's broken. What's needed is a patch.

regards, tom lane

On Jun 15 09:33, Tom Lane wrote:

Volkan YAZICI <yazicivo@ttnet.net.tr> writes:

Also, if you'd wish, I can prepare an ad-hoc regression tests patch
for LATIN5 and UTF-8 support of Turkish characters.

We know it's broken. What's needed is a patch.

I couldn't understand why you're so aggressive. I'm just trying to help.
And, IMNSHO, posted test results are quite helpful to determine the
exact problem.

As I understand from the tests, ILIKE and ~* don't work properly while
using UTF-8, despite lower() and upper() work without any problem.
Therefore, I've tried to imitate the code of lower() to form a working
iwchareq() method. [Related patch is attached.] It succeded in all of my
previous tests (and plus in regression tests).

As you can see, it's a quite ad-hoc patch. (No win32 support added yet.)
Also, it needs a HAVE_MBTOWC definition too. I just wanted to give it a
V0 speed.

I think, like.c and oracle_compat.c files should be written from
scratch by somebody with more experience. They look like deprecated in
some aspects. (For instance, like.c is still using CHARMAX despite Bruce
generalized it as HIGHBIT in c.h)

Regards.

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

Like the bug report that was submitted a few days ago,
http://archives.postgresql.org/pgsql-bugs/2005-09/msg00233.php
I have the same thing for Turkish locale.

Would you confirm that this is fixed by my recent commit?

regards, tom lane

Hello Tom,

On Mon, 2006-09-04 at 14:35 -0400, Tom Lane wrote:

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

Like the bug report that was submitted a few days ago,
http://archives.postgresql.org/pgsql-bugs/2005-09/msg00233.php
I have the same thing for Turkish locale.

Would you confirm that this is fixed by my recent commit?

If I did not miss any test; I can confirm that this bug is fixed for
Turkish locale. I does return the correct results for each query, each
special char.

Thanks Tom.

Regards,
--
The PostgreSQL Company - Command Prompt, Inc. 1.503.667.4564
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

On Sep 04 02:35, Tom Lane wrote:

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

Like the bug report that was submitted a few days ago,
http://archives.postgresql.org/pgsql-bugs/2005-09/msg00233.php
I have the same thing for Turkish locale.

Would you confirm that this is fixed by my recent commit?

I've made some tests. [AFAICS, below results are applicable for all
latinN encodings.]

Cluster Locale | client_encoding | upper() | lower() | ILIKE | ~*
-----------------+-----------------+---------+---------+-------+----
tr_TR.iso8859-9 | LATIN5 | OK | OK | OK | OK
tr_TR.iso8859-9 | UTF8 | OK | OK | OK | OK
tr_TR.UTF-8 | LATIN5 | OK | OK | OK | FAILS
tr_TR.UTF-8 | UTF8 | OK | OK | OK | FAILS

Looks like regex code needs same modifications done to ILIKE command.

Regards.

Volkan YAZICI <yazicivo@ttnet.net.tr> writes:

On Sep 04 02:35, Tom Lane wrote:

"Devrim GUNDUZ" <devrim@gunduz.org> writes:

Like the bug report that was submitted a few days ago,
http://archives.postgresql.org/pgsql-bugs/2005-09/msg00233.php
I have the same thing for Turkish locale.

Would you confirm that this is fixed by my recent commit?

I've made some tests. [AFAICS, below results are applicable for all
latinN encodings.]

Cluster Locale | client_encoding | upper() | lower() | ILIKE | ~*
-----------------+-----------------+---------+---------+-------+----
tr_TR.iso8859-9 | LATIN5 | OK | OK | OK | OK
tr_TR.iso8859-9 | UTF8 | OK | OK | OK | OK
tr_TR.UTF-8 | LATIN5 | OK | OK | OK | FAILS
tr_TR.UTF-8 | UTF8 | OK | OK | OK | FAILS

Looks like regex code needs same modifications done to ILIKE command.

I have applied the patch for like.c: r 1.64 -> r 1.65, to my
postgresql 8.1.4 sources, and ILIKE now, obviously, works for
ru_RU.UTF-8 locale.

I still have a couple of issues, though, not sure if anyone reported
them yet.

1) When table or column names are multibyte, they are not displayed by
either psql or pgAdmin. What I see is empty strings instead of
table names. It works with latin table names and with national
table names on single-byte encodings.

2) When I try to create a stored procedure to create table (e.g., for
creating table only if it doesn't already exist), it fails to
compile if i use cyrillic letter "х" (unicode id: 0x0445, utf-8
representation is D1 85) with the following weird error:

ERROR: expected dot between identifiers: х
CONTEXT: compile of PL/pgSQL function "createoraltertable" near line 2

the following query fails:

CREATE OR REPLACE FUNCTION TestFunction()
RETURNS int
AS $$
BEGIN
SELECT х FROM test_table;
RETURN 0;
END;
$$ LANGUAGE plpgsql;

(the only multibyte symbol is that 0x0445 letter, and it's between
select and from).

Simple create table and select, outside of function, works (except the
first issue).

What can I do to help to debug this?
Should I try the sources from CVS HEAD?

--
WBR, Victor V. Snezhko
E-mail: snezhko@indorsoft.ru

Victor Snezhko <snezhko@indorsoft.ru> writes:

2) When I try to create a stored procedure to create table (e.g., for
creating table only if it doesn't already exist), it fails to
compile if i use cyrillic letter "х" (unicode id: 0x0445, utf-8
representation is D1 85) with the following weird error:

ERROR: expected dot between identifiers: х
CONTEXT: compile of PL/pgSQL function "createoraltertable" near line 2

the following query fails:

CREATE OR REPLACE FUNCTION TestFunction()
RETURNS int
AS $$
BEGIN
SELECT х FROM test_table;
RETURN 0;
END;
$$ LANGUAGE plpgsql;

Sorry, I was a bit sloppy here: I said about creating table and
specified error output from function createoraltertable(),
which contained CREATE TABLE statement instead of SELECT, but
specified another example function.

Clarification: TestFunction() also fails with the same error about
expected dot between identifiers and can be used for debugging.

--
WBR, Victor V. Snezhko
E-mail: snezhko@indorsoft.ru

Volkan YAZICI <yazicivo@ttnet.net.tr> writes:

Looks like regex code needs same modifications done to ILIKE command.

Regex is still broken but I don't think the same fix is feasible -- we
can't blindly downcase the strings outside the regex package (consider
embedded ARE options). And even if we did that, it would fix only one
aspect of regex's locale dependence. The only real fix is to
de-lobotomize the locale-specific functions in regc_locale.c, and then
you run up against the same problem that pg_wchar may or may not use the
same representation that the <wchar.h> functions expect. So I fear
regex will have to stay as it is until we have our own locale support.

regards, tom lane