tsearch2 anomoly?

This is how default parser works. See output from
select * from ts_debug('gallery2-httpd-conf');
and
select * from ts_debug('httpd-2.2.3-5.src.rpm');

All token type:

select * from token_type();

On Thu, 6 Sep 2007, RC Gobeille wrote:

I'm having trouble understanding to_tsvector. (PostreSQL 8.1.9 contrib)

In this first case converting 'gallery2-httpd-conf' makes sense to me and is
exactly what I want. It looks like the entire string is indexed plus the
substrings broken by '-' are indexed.

ossdb=# select to_tsvector('gallery2-httpd-conf');
to_tsvector
---------------------------------------------------------
'conf':4 'httpd':3 'gallery2':2 'gallery2-httpd-conf':1

However, I'd expect the same to happen in the httpd example - but it does not
appear to.

ossdb=# select to_tsvector('httpd-2.2.3-5.src.rpm');
to_tsvector
---------------------------
'httpd-2.2.3-5.src.rpm':1

Why don't I get: 'httpd', 'src', 'rpm', 'httpd-2.2.3-5.src.rpm' ?

Is this a bug or design?

Thank you!
Bob

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Thanks and I didn't know about ts_debug, so thanks for that also.

For the record, I see how to use my own processing function (e.g.
dropatsymbol) to get what I need:
http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/tsearch-V2-intro
.html

However, can you explain the logic behind the parsing difference if I just
add a ".s" to a string:

ossdb=# select ts_debug('gallery2-httpd-2.1-conf.');
ts_debug
-----------------------------------------------------------------------
(default,hword,"Hyphenated word",gallery2-httpd-2,{simple},"'2' 'httpd'
'gallery2' 'gallery2-httpd-2'")
(default,part_hword,"Part of hyphenated word",gallery2,{simple},'gallery2')
(default,lpart_hword,"Latin part of hyphenated
word",httpd,{en_stem},'httpd')
(default,float,"Decimal notation",2.1,{simple},'2.1')
(default,lpart_hword,"Latin part of hyphenated word",conf,{en_stem},'conf')
(5 rows)

ossdb=# select ts_debug('gallery2-httpd-2.1-conf.s');
ts_debug
---------------------------------------------------------------------
(default,host,Host,gallery2-httpd-2.1-conf.s,{simple},'gallery2-httpd-2.1-c
onf.s')
(1 row)

Thanks again,
Bob

On 9/6/07 11:19 AM, "Oleg Bartunov" <oleg@sai.msu.su> wrote:

Usual text hasn't strict syntax rules, so parser tries to recognize most
probable token. Something with '.', '-' and alnum characters is often a
filename, but filename is very rare finished or started by dot.

RC Gobeille wrote:

Thanks and I didn't know about ts_debug, so thanks for that also.

For the record, I see how to use my own processing function (e.g.
dropatsymbol) to get what I need:
http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/tsearch-V2-intro
.html

However, can you explain the logic behind the parsing difference if I just
add a ".s" to a string:

ossdb=# select ts_debug('gallery2-httpd-2.1-conf.');
ts_debug
-----------------------------------------------------------------------
(default,hword,"Hyphenated word",gallery2-httpd-2,{simple},"'2' 'httpd'
'gallery2' 'gallery2-httpd-2'")
(default,part_hword,"Part of hyphenated word",gallery2,{simple},'gallery2')
(default,lpart_hword,"Latin part of hyphenated
word",httpd,{en_stem},'httpd')
(default,float,"Decimal notation",2.1,{simple},'2.1')
(default,lpart_hword,"Latin part of hyphenated word",conf,{en_stem},'conf')
(5 rows)

ossdb=# select ts_debug('gallery2-httpd-2.1-conf.s');
ts_debug
---------------------------------------------------------------------
(default,host,Host,gallery2-httpd-2.1-conf.s,{simple},'gallery2-httpd-2.1-c
onf.s')
(1 row)

Thanks again,
Bob

On 9/6/07 11:19 AM, "Oleg Bartunov" <oleg@sai.msu.su> wrote:

This is how default parser works. See output from
select * from ts_debug('gallery2-httpd-conf');
and
select * from ts_debug('httpd-2.2.3-5.src.rpm');

All token type:

select * from token_type();

On Thu, 6 Sep 2007, RC Gobeille wrote:

I'm having trouble understanding to_tsvector. (PostreSQL 8.1.9 contrib)

In this first case converting 'gallery2-httpd-conf' makes sense to me and is
exactly what I want. It looks like the entire string is indexed plus the
substrings broken by '-' are indexed.

ossdb=# select to_tsvector('gallery2-httpd-conf');
to_tsvector
---------------------------------------------------------
'conf':4 'httpd':3 'gallery2':2 'gallery2-httpd-conf':1

However, I'd expect the same to happen in the httpd example - but it does not
appear to.

ossdb=# select to_tsvector('httpd-2.2.3-5.src.rpm');
to_tsvector
---------------------------
'httpd-2.2.3-5.src.rpm':1

Why don't I get: 'httpd', 'src', 'rpm', 'httpd-2.2.3-5.src.rpm' ?

Is this a bug or design?

Thank you!
Bob

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/