Tsearch2 & Hebrew

Hello Jonatan,

Am 2006-08-30 19:09:19, schrieb Yonatan Ben-Nes:

I want to use Tsearch2 for a current project I have but I can't seem to
find a way to implement it on hebrew content.

I have the same problem since I have an UTF-8 Database of arround
380 GByte (growing 100 MByte per day) in over 60 languages and
can not search in arabic, farsi and hebrew.

It seems, that there is NO solution for those three languages

Greetings
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
50, rue de Soultz MSM LinuxMichi
0033/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)

You need to provide more details.

Oleg
On Fri, 1 Sep 2006, Michelle Konzack wrote:

Hello Jonatan,

Am 2006-08-30 19:09:19, schrieb Yonatan Ben-Nes:

I want to use Tsearch2 for a current project I have but I can't seem to
find a way to implement it on hebrew content.

I have the same problem since I have an UTF-8 Database of arround
380 GByte (growing 100 MByte per day) in over 60 languages and
can not search in arabic, farsi and hebrew.

It seems, that there is NO solution for those three languages

Greetings
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Hello Oleg,

Am 2006-09-04 20:52:02, schrieb Oleg Bartunov:

You need to provide more details.

Oleg

------------------------- END OF REPLIED MESSAGE -------------------------

One of my two programmers had coded last year stuff in php5 (UNICODE
is now working) to search � la Google in my Database. I am collecting
international stuff about wars, warcrime and violation of human rights.

My database is text/plain UNICODE and has curently arround 380-390 GB
which I have splited into tables of 10 years...

It seems, there is a problem with BIDI searching. Russian and chinese
is NO problem. Many texts are mixed like US-ASCII, arabic and hebrew.

Now if I enter search strings it returns nothing.
Even if I am in psql with multilingual terminal or in pgadmin.

So it can not be a problem with PHP5.

Oh yes, since I have switched to one table per 10 years, tsearch2 do
not want to search my whole Database... but for tsearch2 I think, I
am looking for a PHP5/PGSQL coder on <www.getacoder.com> since I am
not the master of PHP5 and PGSQL. (I am more Sysadmin and soldier
then programmer even if I can code stuff)

Greetings
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
50, rue de Soultz MSM LinuxMichi
0033/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)

Hi all,

Well my problem was that I didn't know if Tsearch2 can work on hebrew
data without a fitting stemmer, my current solution is to use the
'simple' dictionary so no lexem is returned.
I wonder if there is an hebrew stemmer which I can use but I can't seem
to find one, so sadly one of the best features of Tsearch2 isn't working
for me.

If I'm wrong please let me know :)

Thanks a lot in advance,

Yonatan Ben-Nes

Oleg Bartunov wrote:

On Tue, 5 Sep 2006, Yonatan Ben-Nes wrote:

Hi all,

Well my problem was that I didn't know if Tsearch2 can work on hebrew data
without a fitting stemmer, my current solution is to use the 'simple'
dictionary so no lexem is returned.
I wonder if there is an hebrew stemmer which I can use but I can't seem to
find one, so sadly one of the best features of Tsearch2 isn't working for me.

Do you use hebrew ispell dictionary ?

If I'm wrong please let me know :)

Thanks a lot in advance,

Yonatan Ben-Nes

Oleg Bartunov wrote:

You need to provide more details.

Oleg
On Fri, 1 Sep 2006, Michelle Konzack wrote:

Hello Jonatan,

Am 2006-08-30 19:09:19, schrieb Yonatan Ben-Nes:

I want to use Tsearch2 for a current project I have but I can't seem to
find a way to implement it on hebrew content.

I have the same problem since I have an UTF-8 Database of arround
380 GByte (growing 100 MByte per day) in over 60 languages and
can not search in arabic, farsi and hebrew.

It seems, that there is NO solution for those three languages

Greetings
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

__________ NOD32 1.1739 (20060904) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

No, I didn't thought that it will be useful if it won't be accompanied
by an hebrew stemmer which will work with it... I'm wrong?

Oleg Bartunov wrote:

On Tue, 5 Sep 2006, Yonatan Ben-Nes wrote:

No, I didn't thought that it will be useful if it won't be accompanied by an
hebrew stemmer which will work with it... I'm wrong?

ispell and stemmer are doing the same job, so you may use

ispell,simple configuration instead of "ideal" one: ispell, stemmer

Of course, some words will not recognized and will leave as is.
Also, you may write very simple stemmer using collection of very common
endings.

Oleg

Oleg Bartunov wrote:

On Tue, 5 Sep 2006, Yonatan Ben-Nes wrote:

Hi all,

Well my problem was that I didn't know if Tsearch2 can work on hebrew data
without a fitting stemmer, my current solution is to use the 'simple'
dictionary so no lexem is returned.
I wonder if there is an hebrew stemmer which I can use but I can't seem to
find one, so sadly one of the best features of Tsearch2 isn't working for
me.

Do you use hebrew ispell dictionary ?

If I'm wrong please let me know :)

Thanks a lot in advance,

Yonatan Ben-Nes

Oleg Bartunov wrote:

You need to provide more details.

Oleg
On Fri, 1 Sep 2006, Michelle Konzack wrote:

Hello Jonatan,

Am 2006-08-30 19:09:19, schrieb Yonatan Ben-Nes:

I want to use Tsearch2 for a current project I have but I can't seem to
find a way to implement it on hebrew content.

I have the same problem since I have an UTF-8 Database of arround
380 GByte (growing 100 MByte per day) in over 60 languages and
can not search in arabic, farsi and hebrew.

It seems, that there is NO solution for those three languages

Greetings
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

http://archives.postgresql.org

__________ NOD32 1.1739 (20060904) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

__________ NOD32 1.1739 (20060904) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:

On Tue, 5 Sep 2006, Yonatan Ben-Nes wrote:

No, I didn't thought that it will be useful if it won't be
accompanied by an hebrew stemmer which will work with it... I'm wrong?

ispell and stemmer are doing the same job, so you may use

ispell,simple configuration instead of "ideal" one: ispell, stemmer

Of course, some words will not recognized and will leave as is.
Also, you may write very simple stemmer using collection of very common
endings.

Oleg

Thanks a lot I'll do that at once!
Yonatan Ben-Nes