New PostgreSQL search resource

Ok, this is now the second site that has come to be in the last few days that
is using FTS and Tsearch for site search, and providing something that seems
a lot better than the search available on the main website... what is
stopping us from implementing this and dumping mnogosearch? at least for the
main site if not for the archives?

Robert Treat

On Friday 16 January 2004 19:33, Joshua D. Drake wrote:

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

Sincerely,

Joshua Drake

--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

On Fri, 16 Jan 2004, Robert Treat wrote:

Ok, this is now the second site that has come to be in the last few days that
is using FTS and Tsearch for site search, and providing something that seems
a lot better than the search available on the main website... what is
stopping us from implementing this and dumping mnogosearch? at least for the
main site if not for the archives?

actually, Dave is working on an improved search ... but, Oleg just
announced a crawler using tsearch that I'm going to take a look at
implementing as well ...

Robert Treat

On Friday 16 January 2004 19:33, Joshua D. Drake wrote:

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

Sincerely,

Joshua Drake

--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

On Fri, 16 Jan 2004, Joshua D. Drake wrote:

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

I'd appreciate if you mention somewhere OpenFTS utilization, so people
could recognize it. But I'm not insisting ;)

Sincerely,

Joshua Drake

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

http://www.commandprompt.com/community/

I'd appreciate if you mention somewhere OpenFTS utilization, so people
could recognize it. But I'm not insisting ;)

Full intent to do so... want to get it finished up first. :)

Sincerely,

Joshua Drake

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL

On Fri, 16 Jan 2004, Marc G. Fournier wrote:

On Fri, 16 Jan 2004, Robert Treat wrote:

Ok, this is now the second site that has come to be in the last few days that
is using FTS and Tsearch for site search, and providing something that seems
a lot better than the search available on the main website... what is
stopping us from implementing this and dumping mnogosearch? at least for the
main site if not for the archives?

actually, Dave is working on an improved search ... but, Oleg just
announced a crawler using tsearch that I'm going to take a look at
implementing as well ...

I suggest to follow Roberts suggestion about indexing main site.
Archives is a bit another thing. I already wrote it needs to be
optimized for crawlers (headers).

Also, It would be much better to be able to index just content without
headers/footers etc. If you have access to files :), you may index
much faster without any crawler ! OpenFTS distribution contains
example scripts to index file collections. It's very very easy.

I quoting from "Survival Guide":

APOD collection is consists of 1757 articles (about 7 Mb) and ideally suited
for OpenFTS. Indexing tooks about 29 seconds on my IBM ThinkPad T21 notebook
( Linux, 2.4.17, 256 Mb RAM, 20 Gb IDE HD). Total number of lexems is
131310, while the number of unique lexemes is only 8,806
( using Porter's stemmer ).

Official PostgreSQL documentation is about the same size.

Robert Treat

On Friday 16 January 2004 19:33, Joshua D. Drake wrote:

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

Sincerely,

Joshua Drake

--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Is it possible to institute phrase searching with that system?

I.e.:

"create unique index"

"select distinct"

etc...

???

On Fri, 16 Jan 2004, Joshua D. Drake wrote:

Scott,

"create unique index"

"select distinct"

I believe that one does this by adding to the word root dictionary and
rebuilding the index. The problem is that then anything indexed as "select
distinct" shows up under searches for "select distinct" but not for
"distinct".

Try searching on SELECT & DISTINCT

--
-Josh Berkus
Aglio Database Solutions
San Francisco

scott.marlowe wrote:

Is it possible to institute phrase searching with that system?

It seems to work fine???

Sincerely,

Joshua D. Drake

I.e.:

"create unique index"

"select distinct"

etc...

???

On Fri, 16 Jan 2004, Joshua D. Drake wrote:

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

Sincerely,

Joshua Drake

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

On Mon, 19 Jan 2004, Joshua D. Drake wrote:

scott.marlowe wrote:

Is it possible to institute phrase searching with that system?

It seems to work fine???

Sincerely,

Joshua D. Drake

Not really. with htdig, I can do "select distinct" on the 7.3 docs and I
get exactly 7 responses back, and they all have select followed
immediately by distinct, with only space, CR / LF etc in between.

And the & symbol josh recommended shows as being ignored, so that didn't
work either.

Word combos like "bulk loads" or "run-time configuration" are quite handy
filters.

Not really. with htdig, I can do "select distinct" on the 7.3 docs and I
get exactly 7 responses back, and they all have select followed
immediately by distinct, with only space, CR / LF etc in between.

Well, if you search for select distinct... OpenFTS automatically searches
for select & distinct. My search returned:

5 results with the first one being this:

http://www.commandprompt.com/community//search.lxp?lxpwrap=734/queries-select-lists.html

Which seems fairly accurate ;). Perhaps a better description of what the
problem actually is or
the behavior you are expecting?

SIncerely,

Joshua D. Drake

And the & symbol josh recommended shows as being ignored, so that didn't
work either.

Word combos like "bulk loads" or "run-time configuration" are quite handy
filters.

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

I think most of the people use http://www.postgresql.org/search.cgi

Default settings, search for select distinct returns in about 10
minutes, with "You haven't typed any word(s) to search for."

Same query with match full phrase... (I did not wait long enough. But
several minutes in any case. Don't know about the result.)

Same query with match full phrase and 7.4 documentation. 2 results in 5
seconds. Results are select into and pg_select.

Could we do something to this search? (Change the defaults, divide it to
smaller parts, use different search engine...)

Rgs,

Jussi

Joshua D. Drake wrote:

Jussi Mikkola wrote:

I think most of the people use http://www.postgresql.org/search.cgi

Yep... Supposedly it is being worked on to make it faster.

J

Default settings, search for select distinct returns in about 10
minutes, with "You haven't typed any word(s) to search for."

Same query with match full phrase... (I did not wait long enough. But
several minutes in any case. Don't know about the result.)

Same query with match full phrase and 7.4 documentation. 2 results in
5 seconds. Results are select into and pg_select.

Could we do something to this search? (Change the defaults, divide it
to smaller parts, use different search engine...)

Rgs,

Jussi

Joshua D. Drake wrote:

Not really. with htdig, I can do "select distinct" on the 7.3 docs
and I get exactly 7 responses back, and they all have select
followed immediately by distinct, with only space, CR / LF etc in
between.

Well, if you search for select distinct... OpenFTS automatically
searches
for select & distinct. My search returned:

5 results with the first one being this:

http://www.commandprompt.com/community//search.lxp?lxpwrap=734/queries-select-lists.html

Which seems fairly accurate ;). Perhaps a better description of what
the problem actually is or
the behavior you are expecting?

SIncerely,

Joshua D. Drake

And the & symbol josh recommended shows as being ignored, so that
didn't work either.

Word combos like "bulk loads" or "run-time configuration" are quite
handy filters.

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

On Mon, 19 Jan 2004, Joshua D. Drake wrote:

Well, if you search for select distinct... OpenFTS automatically searches
for select & distinct. My search returned:

5 results with the first one being this:

http://www.commandprompt.com/community//search.lxp?lxpwrap=734/queries-select-lists.html

Which seems fairly accurate ;). Perhaps a better description of what the
problem actually is or
the behavior you are expecting?

Yes, phrase searching generally means you enter an exact string of words,
in a particular order, and it searches on those strings in that order.

this comes in handy when looking for strings where the words together mean
one thing, and apart may mean something else, like

set transaction isolation level

Many of those words appear throughout the docs, and may well all appear on
the same page but not be tied together with each other on each page.

If we do a phrase search we only see the places where those words are
together. Individual word searching gives me 11 results in htdig, whereas
the phrase gives me 3 hits, the second of which is obviously the correct
one (the first is release notes for 6.5... hmmm.) With the 11 results,
moves it down to number 7.

This is especially useful when you can search on a couple of small phrases
that might go together. Anyway, it's no big deal, just wondering if it
supported phrase searching.