Problem with site doc search

On Sat, 2 Feb 2008, Gurjeet Singh wrote:

Hi All,

I just noticed a minor bug in our search results. Searching for
is_insteadbool in 8.3 docs returns the following page:

http://www.postgresql.org/docs/8.3/static/catalog-pg-rewrite.html

is_instead is a column, and bool is the datatype, both mentioned in
different columns. I know it is based on postgres' own full text search, but
am not sure about the method how docs are read by the search engine. It
seems that the problem lies in the the way the doc was read and fed to the
index builder, because the initial search results show these two words
combined.

Also, is it possible to teach our search engine to *not* treat _
(underscore) as a word separator? This would be great help and would result
much better results.

Sure, there are many ways to do this. Magnus ?

Thanks and best regards,

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:

On Sat, 2 Feb 2008, Gurjeet Singh wrote:

Hi All,

I just noticed a minor bug in our search results. Searching for
is_insteadbool in 8.3 docs returns the following page:

http://www.postgresql.org/docs/8.3/static/catalog-pg-rewrite.html

is_instead is a column, and bool is the datatype, both mentioned in
different columns. I know it is based on postgres' own full text
search, but
am not sure about the method how docs are read by the search engine. It
seems that the problem lies in the the way the doc was read and fed to
the
index builder, because the initial search results show these two words
combined.

Also, is it possible to teach our search engine to *not* treat _
(underscore) as a word separator? This would be great help and would
result
much better results.

Sure, there are many ways to do this. Magnus ?

Which way would you recommend? You're the tsearch master ;-)

//Magnus

Hi guys.... any updates on this? Pinging you just so that we do not forget
it in the heap of mails in our inboxes.

Best regards,

On Feb 3, 2008 8:40 AM, Magnus Hagander <magnus@hagander.net> wrote:

Oleg Bartunov wrote:

On Sat, 2 Feb 2008, Gurjeet Singh wrote:

Hi All,

I just noticed a minor bug in our search results. Searching for
is_insteadbool in 8.3 docs returns the following page:

http://www.postgresql.org/docs/8.3/static/catalog-pg-rewrite.html

is_instead is a column, and bool is the datatype, both mentioned in
different columns. I know it is based on postgres' own full text
search, but
am not sure about the method how docs are read by the search engine. It
seems that the problem lies in the the way the doc was read and fed to
the
index builder, because the initial search results show these two words
combined.

Also, is it possible to teach our search engine to *not* treat _
(underscore) as a word separator? This would be great help and would
result
much better results.

Sure, there are many ways to do this. Magnus ?

Which way would you recommend? You're the tsearch master ;-)

//Magnus

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

17° 29' 34.37"N, 78° 30' 59.76"E - Hyderabad
18° 32' 57.25"N, 73° 56' 25.42"E - Pune
37° 47' 19.72"N, 122° 24' 1.69" W - San Francisco *

http://gurjeet.frihost.net

Mail sent from my BlackLaptop device

No. It's on the list, but other things around the release haev priority.

//Magnus

On Tue, 5 Feb 2008, Magnus Hagander wrote:

No. It's on the list, but other things around the release haev priority.

I just returned from my Europe trip and have many things to do :)

//Magnus

On Mon, Feb 04, 2008 at 06:43:09PM -0800, Gurjeet Singh wrote:

Hi guys.... any updates on this? Pinging you just so that we do not forget
it in the heap of mails in our inboxes.

Best regards,

On Feb 3, 2008 8:40 AM, Magnus Hagander <magnus@hagander.net> wrote:

Oleg Bartunov wrote:

On Sat, 2 Feb 2008, Gurjeet Singh wrote:

Hi All,

I just noticed a minor bug in our search results. Searching for
is_insteadbool in 8.3 docs returns the following page:

http://www.postgresql.org/docs/8.3/static/catalog-pg-rewrite.html

is_instead is a column, and bool is the datatype, both mentioned in
different columns. I know it is based on postgres' own full text
search, but
am not sure about the method how docs are read by the search engine. It
seems that the problem lies in the the way the doc was read and fed to
the
index builder, because the initial search results show these two words
combined.

Also, is it possible to teach our search engine to *not* treat _
(underscore) as a word separator? This would be great help and would
result
much better results.

Sure, there are many ways to do this. Magnus ?

Which way would you recommend? You're the tsearch master ;-)

//Magnus

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

17° 29' 34.37"N, 78° 30' 59.76"E - Hyderabad
18° 32' 57.25"N, 73° 56' 25.42"E - Pune
37° 47' 19.72"N, 122° 24' 1.69" W - San Francisco *

http://gurjeet.frihost.net

Mail sent from my BlackLaptop device

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Has this been addressed?

---------------------------------------------------------------------------

Oleg Bartunov wrote:

On Tue, 5 Feb 2008, Magnus Hagander wrote:

No. It's on the list, but other things around the release haev priority.

I just returned from my Europe trip and have many things to do :)

//Magnus

On Mon, Feb 04, 2008 at 06:43:09PM -0800, Gurjeet Singh wrote:

Hi guys.... any updates on this? Pinging you just so that we do not forget
it in the heap of mails in our inboxes.

Best regards,

On Feb 3, 2008 8:40 AM, Magnus Hagander <magnus@hagander.net> wrote:

Oleg Bartunov wrote:

On Sat, 2 Feb 2008, Gurjeet Singh wrote:

Hi All,

I just noticed a minor bug in our search results. Searching for
is_insteadbool in 8.3 docs returns the following page:

http://www.postgresql.org/docs/8.3/static/catalog-pg-rewrite.html

is_instead is a column, and bool is the datatype, both mentioned in
different columns. I know it is based on postgres' own full text
search, but
am not sure about the method how docs are read by the search engine. It
seems that the problem lies in the the way the doc was read and fed to
the
index builder, because the initial search results show these two words
combined.

Also, is it possible to teach our search engine to *not* treat _
(underscore) as a word separator? This would be great help and would
result
much better results.

Sure, there are many ways to do this. Magnus ?

Which way would you recommend? You're the tsearch master ;-)

//Magnus

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

17? 29' 34.37"N, 78? 30' 59.76"E - Hyderabad
18? 32' 57.25"N, 73? 56' 25.42"E - Pune
37? 47' 19.72"N, 122? 24' 1.69" W - San Francisco *

http://gurjeet.frihost.net

Mail sent from my BlackLaptop device

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

No, it's still on the TODO. Gevik has also been looking a bit at it (I
think - at least he's indicated that he is), and he recently got some nwe
parser code to look at to see if we can use to fix it.

//Magnus

No, it's still on the TODO. Gevik has also been looking a bit
at it (I think - at least he's indicated that he is), and he
recently got some new parser code to look at to see if we can
use to fix it.

I have the new parser code. Next week I have some time reserved to look at
it.

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl> wrote:

No, it's still on the TODO. Gevik has also been looking a bit
at it (I think - at least he's indicated that he is), and he
recently got some new parser code to look at to see if we can
use to fix it.

I have the new parser code. Next week I have some time reserved to look at
it.

For the last few days I have been noticing that the _ (underscore) is no
longer being treated as a word boundary. Can you confirm this has been
fixed?

Best regards,

--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

Gurjeet Singh wrote:

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl
<mailto:pgdev@xs4all.nl>> wrote:

No, it's still on the TODO. Gevik has also been looking a bit
at it (I think - at least he's indicated that he is), and he
recently got some new parser code to look at to see if we can
use to fix it.

I have the new parser code. Next week I have some time reserved to
look at
it.

For the last few days I have been noticing that the _ (underscore) is no
longer being treated as a word boundary. Can you confirm this has been
fixed?

AFAIK, nothing has been done yet :-)

//Magnus

On Tue, Mar 25, 2008 at 3:01 AM, Magnus Hagander <magnus@hagander.net>
wrote:

Gurjeet Singh wrote:

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl
<mailto:pgdev@xs4all.nl>> wrote:

No, it's still on the TODO. Gevik has also been looking a bit
at it (I think - at least he's indicated that he is), and he
recently got some new parser code to look at to see if we can
use to fix it.

I have the new parser code. Next week I have some time reserved to
look at
it.

For the last few days I have been noticing that the _ (underscore) is no
longer being treated as a word boundary. Can you confirm this has been
fixed?

AFAIK, nothing has been done yet :-)

You are right.... I was looking for pg_standby and another pg_<something>
and since I found those results on top, I thought there was some action
taken.

The problem persists....

Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

I just searched for pg_standby and it looks like it is fixed now.

---------------------------------------------------------------------------

Gurjeet Singh wrote:

On Tue, Mar 25, 2008 at 3:01 AM, Magnus Hagander <magnus@hagander.net>
wrote:

Gurjeet Singh wrote:

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl
<mailto:pgdev@xs4all.nl>> wrote:

No, it's still on the TODO. Gevik has also been looking a bit
at it (I think - at least he's indicated that he is), and he
recently got some new parser code to look at to see if we can
use to fix it.

I have the new parser code. Next week I have some time reserved to
look at
it.

For the last few days I have been noticing that the _ (underscore) is no
longer being treated as a word boundary. Can you confirm this has been
fixed?

AFAIK, nothing has been done yet :-)

You are right.... I was looking for pg_standby and another pg_<something>
and since I found those results on top, I thought there was some action
taken.

The problem persists....

Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

I didn't do anything, but possibly it got fixed by a different upgrade
at some point, and the recrawling of the sites.

//Magnus

Bruce Momjian wrote:

Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different upgrade
at some point, and the recrawling of the sites.

Oops, maybe it isn't fixed. I tried "pg_standby" and it seemed to work
but "pg" and "standby" returns the same results.

Is that correct? How do I test this?

---------------------------------------------------------------------------

//Magnus

Bruce Momjian wrote:

I just searched for pg_standby and it looks like it is fixed now.

---------------------------------------------------------------------------

Gurjeet Singh wrote:

On Tue, Mar 25, 2008 at 3:01 AM, Magnus Hagander
<magnus@hagander.net> wrote:

Gurjeet Singh wrote:

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl
<mailto:pgdev@xs4all.nl>> wrote:

No, it's still on the TODO. Gevik has also been looking
a bit at it (I think - at least he's indicated that he
is), and he recently got some new parser code to look at
to see if we can use to fix it.

I have the new parser code. Next week I have some time
reserved to look at
it.

For the last few days I have been noticing that the _
(underscore) is no longer being treated as a word boundary. Can
you confirm this has been fixed?

AFAIK, nothing has been done yet :-)

You are right.... I was looking for pg_standby and another
pg_<something> and since I found those results on top, I thought
there was some action taken.

The problem persists....

Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different upgrade
at some point, and the recrawling of the sites.

Oops, maybe it isn't fixed. I tried "pg_standby" and it seemed to work
but "pg" and "standby" returns the same results.

Is that correct? How do I test this?

The default is to split words on underscore, so it's probably doing what
it always did. Try "to_tsquery" and you should see it matching "tsquery"
(probably "to" is a stopword).

I did put together a custom parser that allowed underscore in words, but
given my extensive "C" experience in the last decade (one tsearch
parser) you don't want to just plug that into the live site. Someone
(Gevik?) was going to have a look at it when they had the time, but I'd
guess that's the one thing none of us have much of.

--
Richard Huxton
Archonet Ltd

Richard Huxton wrote:

Bruce Momjian wrote:

Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different
upgrade at some point, and the recrawling of the sites.

Oops, maybe it isn't fixed. I tried "pg_standby" and it seemed to
work but "pg" and "standby" returns the same results.

Is that correct? How do I test this?

The default is to split words on underscore, so it's probably doing
what it always did. Try "to_tsquery" and you should see it matching
"tsquery" (probably "to" is a stopword).

I did put together a custom parser that allowed underscore in words,
but given my extensive "C" experience in the last decade (one tsearch
parser) you don't want to just plug that into the live site. Someone
(Gevik?) was going to have a look at it when they had the time, but
I'd guess that's the one thing none of us have much of.

Yeah.
Did you ever post the code to anybody other than Gevik? If not, please
send it to pgsql-www and someone can give it a quick look-over (perhaps
Oleg can help us there?)

//Magnus

Magnus Hagander wrote:

Did you ever post the code to anybody other than Gevik? If not, please
send it to pgsql-www and someone can give it a quick look-over (perhaps
Oleg can help us there?)

Will do.

--
Richard Huxton
Archonet Ltd

On Tue, 15 Apr 2008, Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different upgrade
at some point, and the recrawling of the sites.

Magnus, we have parser for indexing pgdocs, do you need it ?

//Magnus

Bruce Momjian wrote:

I just searched for pg_standby and it looks like it is fixed now.

---------------------------------------------------------------------------

Gurjeet Singh wrote:

On Tue, Mar 25, 2008 at 3:01 AM, Magnus Hagander
<magnus@hagander.net> wrote:

Gurjeet Singh wrote:

On Thu, Mar 6, 2008 at 5:01 PM, Gevik Babakhani <pgdev@xs4all.nl
<mailto:pgdev@xs4all.nl>> wrote:

No, it's still on the TODO. Gevik has also been looking
a bit at it (I think - at least he's indicated that he
is), and he recently got some new parser code to look at
to see if we can use to fix it.

I have the new parser code. Next week I have some time
reserved to look at
it.

For the last few days I have been noticing that the _
(underscore) is no longer being treated as a word boundary. Can
you confirm this has been fixed?

AFAIK, nothing has been done yet :-)

You are right.... I was looking for pg_standby and another
pg_<something> and since I found those results on top, I thought
there was some action taken.

The problem persists....

Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com

EnterpriseDB http://www.enterprisedb.com

Mail sent from my BlackLaptop device

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Sergey Karpov prepared contrib/extend_parser, which we intend to use for
indexing pg-related documents. It handles '_' properly, so if anybody
interested, we could post it. Also, it can be useful for playing, since
it's standalone contrib module.

Oleg

On Tue, 15 Apr 2008, Richard Huxton wrote:

Bruce Momjian wrote:

Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different upgrade
at some point, and the recrawling of the sites.

Oops, maybe it isn't fixed. I tried "pg_standby" and it seemed to work
but "pg" and "standby" returns the same results.

Is that correct? How do I test this?

The default is to split words on underscore, so it's probably doing what it
always did. Try "to_tsquery" and you should see it matching "tsquery"
(probably "to" is a stopword).

I did put together a custom parser that allowed underscore in words, but
given my extensive "C" experience in the last decade (one tsearch parser) you
don't want to just plug that into the live site. Someone (Gevik?) was going
to have a look at it when they had the time, but I'd guess that's the one
thing none of us have much of.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

Oleg Bartunov wrote:

Sergey Karpov prepared contrib/extend_parser, which we intend to use for
indexing pg-related documents. It handles '_' properly, so if anybody
interested, we could post it. Also, it can be useful for playing, since
it's standalone contrib module.

Does it make sense to back-patch the default parser for 8.4? At present,
it can't handle underscores in file-paths.

--
Richard Huxton
Archonet Ltd

Notice that :

http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=r
and
http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=d

do not provide same result (3 results by date, 1 by rank) even if only the
sorting is changed.

--
Cédric Villemain
Administrateur de Base de Données
Cel: +33 (0)6 74 15 56 53
http://dalibo.com - http://dalibo.org

Oleg Bartunov wrote:

On Tue, 15 Apr 2008, Magnus Hagander wrote:

I didn't do anything, but possibly it got fixed by a different
upgrade at some point, and the recrawling of the sites.

Magnus, we have parser for indexing pgdocs, do you need it ?

Yes, please!

//Magnus

Cédric Villemain wrote:

Notice that :

http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=r
and
http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=d

do not provide same result (3 results by date, 1 by rank) even if
only the sorting is changed.

Actually, I get 5 and 7, in the other order.

The reason for this is that Tom Lane is way too active. It's
gin_fuzzy_search_limit that's doing the restriction first, and the date
restriction comes in later.

//Magnus

Le Wednesday 16 April 2008, Magnus Hagander a écrit :

Cédric Villemain wrote:

Notice that :

http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=r
and
http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=d

do not provide same result (3 results by date, 1 by rank) even if
only the sorting is changed.

Actually, I get 5 and 7, in the other order.

The reason for this is that Tom Lane is way too active. It's
gin_fuzzy_search_limit that's doing the restriction first, and the date
restriction comes in later.

Yes, you are perfectly right. Can I suggest to deactivate
gin_fuzzy_search_limit (or increase the value) when one condition (and use
the condition earlier) can considerably reduce the number of results (like
the 'post date' here) ?

//Magnus

--
Cédric Villemain
Administrateur de Base de Données
Cel: +33 (0)6 74 15 56 53
http://dalibo.com - http://dalibo.org

Cédric Villemain wrote:

Le Wednesday 16 April 2008, Magnus Hagander a écrit :

Cédric Villemain wrote:

Notice that :

http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=r
and
http://search.postgresql.org/search?q=tom+lane&amp;m=1&amp;l=&amp;d=1&amp;s=d

do not provide same result (3 results by date, 1 by rank) even if
only the sorting is changed.

Actually, I get 5 and 7, in the other order.

The reason for this is that Tom Lane is way too active. It's
gin_fuzzy_search_limit that's doing the restriction first, and the
date restriction comes in later.

Yes, you are perfectly right. Can I suggest to deactivate
gin_fuzzy_search_limit (or increase the value) when one condition
(and use the condition earlier) can considerably reduce the number of
results (like the 'post date' here) ?

You'd have to convince the planner to actually not use an indexscan at
all on the tsvector. Normally it'll choose an index scan on each and
then a bitmap join, and we don't want to bring back so many rows...

//Magnus