Include Lists for Text Search
It seems possible to write your own functions to support various
possibilities with text search.
One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.
There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.
Can we include that functionality now?
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.
I don't understand what you're proposing. We already have dict_synonym
that you can use to accept a simple list of words. But that doesn't
change the way GIN and GiST works.
?
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Mon, 10 Sep 2007, Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Mon, 2007-09-10 at 12:58 +0100, Heikki Linnakangas wrote:
Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.I don't understand what you're proposing. We already have dict_synonym
that you can use to accept a simple list of words.
How does that allow me to limit the number of words to a known list?
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.
So there isn't one yet, but you think it will be easy to write and that
we should call it dict_strict?
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 12:58 +0100, Heikki Linnakangas wrote:
Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.I don't understand what you're proposing. We already have dict_synonym
that you can use to accept a simple list of words.How does that allow me to limit the number of words to a known list?
text search doesn't index unknown words, so if your mapping contains
only one dictionary, this dictionary will control what words to index.
While dict_synonym is good for not big list I'd write separate dictionary
with fast lookup.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.So there isn't one yet, but you think it will be easy to write and that
we should call it dict_strict?
we have dict_synonym already and if your list is not big you'll be happy.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
How does that allow me to limit the number of words to a known list?
If all dictionaries returns NULL for token the this token will not be indexed at
all.
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
GIN doesn't depend strongly on number of words. It has log(N) behaviour for
numbers of words because of using B-Tree over words.
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
On Mon, 2007-09-10 at 16:35 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.So there isn't one yet, but you think it will be easy to write and that
we should call it dict_strict?we have dict_synonym already and if your list is not big you'll be happy.
So I need to do something like
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
synonym = include_only_these_words
);
which will then look for a file called include_only_these_words.syn?
I would prefer to be able to do something like this
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
include = justthese
);
...which makes more sense to anyone reading it
and I also want to make the comparison case insensitive.
Would it be better to
1. include a new dictionary file (dict_strict, as you suggest)
2. a) allow case sensitivity as another option in dictionaries
b) allow "include" as another word for "stoplist", but with the
meaning reversed?
e.g.
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
include = justthese,
case_sensitive = true
);
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
On Mon, 2007-09-10 at 16:48 +0400, Teodor Sigaev wrote:
There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
GIN is great, sorry if that sounded negative.
GIN doesn't depend strongly on number of words. It has log(N) behaviour for
numbers of words because of using B-Tree over words.
log(N) in the number of distinct words, but every word you index results
in an index insert, so if we index more words than we need then the
insert rate will go down.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 16:48 +0400, Teodor Sigaev wrote:
There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be lookingGIN is great, sorry if that sounded negative.
GIN doesn't depend strongly on number of words. It has log(N) behaviour for
numbers of words because of using B-Tree over words.log(N) in the number of distinct words, but every word you index results
in an index insert, so if we index more words than we need then the
insert rate will go down.
yes, there is room to improve support of very long posting lists
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 16:35 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
On Mon, 2007-09-10 at 16:10 +0400, Oleg Bartunov wrote:
On Mon, 10 Sep 2007, Simon Riggs wrote:
It seems possible to write your own functions to support various
possibilities with text search.One of the more common thoughts is to have a list of words that you
would like to include, i.e. the opposite of a stop word list.There are clear indications that indexing too many words is a problem
for both GIN and GIST. If people already know what they'll be looking
for and what they will never be looking for, it seems easier to supply
that list up front, rather than hide it behind lots of hand-crafted
code.Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.So there isn't one yet, but you think it will be easy to write and that
we should call it dict_strict?we have dict_synonym already and if your list is not big you'll be happy.
So I need to do something like
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
synonym = include_only_these_words
);which will then look for a file called include_only_these_words.syn?
I would prefer to be able to do something like this
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
include = justthese
);
...which makes more sense to anyone reading it
and I also want to make the comparison case insensitive.Would it be better to
1. include a new dictionary file (dict_strict, as you suggest)
2. a) allow case sensitivity as another option in dictionaries
b) allow "include" as another word for "stoplist", but with the
meaning reversed?e.g.
CREATE TEXT SEARCH DICTIONARY my_diction (
template = snowball,
include = justthese,
case_sensitive = true
);
No, you need to write new template, which efficiently works with
big lists and support case insensitive comparison.
CREATE TEXT SEARCH TEMPLATE biglist (
.....
);
CREATE TEXT SEARCH DICTIONARY my_diction (
TEMPLATE = biglist,
DictFile = words,
case_sensitive = true
);
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov <oleg@sai.msu.su> writes:
On Mon, 10 Sep 2007, Simon Riggs wrote:
Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.
... for 8.4.
regards, tom lane
On Mon, 10 Sep 2007, Tom Lane wrote:
Oleg Bartunov <oleg@sai.msu.su> writes:
On Mon, 10 Sep 2007, Simon Riggs wrote:
Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.... for 8.4.
It can be just a contrib module. There are several useful dictionaries
we need to port.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Mon, 2007-09-10 at 10:21 -0400, Tom Lane wrote:
Oleg Bartunov <oleg@sai.msu.su> writes:
On Mon, 10 Sep 2007, Simon Riggs wrote:
Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.... for 8.4.
I've coded a small patch to allow CaseSensitive synonyms.
CREATE TEXT SEARCH DICTIONARY my_diction (
TEMPLATE = biglist,
DictFile = words,
CaseSensitive = true
);
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
Attachments:
ts_casesensitive.v1.patchtext/x-patch; charset=utf-8; name=ts_casesensitive.v1.patchDownload+23-13
This has been saved for the 8.4 release:
http://momjian.postgresql.org/cgi-bin/pgpatches_hold
---------------------------------------------------------------------------
Simon Riggs wrote:
On Mon, 2007-09-10 at 10:21 -0400, Tom Lane wrote:
Oleg Bartunov <oleg@sai.msu.su> writes:
On Mon, 10 Sep 2007, Simon Riggs wrote:
Can we include that functionality now?
This could be realized very easyly using dict_strict, which returns
only known words, and mapping contains only this dictionary. So,
feel free to write it and submit.... for 8.4.
I've coded a small patch to allow CaseSensitive synonyms.
CREATE TEXT SEARCH DICTIONARY my_diction (
TEMPLATE = biglist,
DictFile = words,
CaseSensitive = true
);--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
[ Attachment, skipping... ]
---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://www.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Simon Riggs <simon@2ndquadrant.com> writes:
I've coded a small patch to allow CaseSensitive synonyms.
Applied with corrections (it'd be good if you at least pretended to test
stuff before submitting it).
Would a similar parameter be useful for any of the other dictionary
types?
regards, tom lane
On Sun, 9 Mar 2008, Tom Lane wrote:
Simon Riggs <simon@2ndquadrant.com> writes:
I've coded a small patch to allow CaseSensitive synonyms.
Applied with corrections (it'd be good if you at least pretended to test
stuff before submitting it).Would a similar parameter be useful for any of the other dictionary
types?
There are many things desirable to do with dictionaries, for example,
say dictionary to return an original word plus it's normal form. Another
feature is a not recognize-and-stop dictionaries, but allow
filtering dictionary. We have a feeling that a little middleware would help
implement this, and CaseSensitive too.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On Sun, 2008-03-09 at 23:03 -0400, Tom Lane wrote:
Simon Riggs <simon@2ndquadrant.com> writes:
I've coded a small patch to allow CaseSensitive synonyms.
Applied with corrections (it'd be good if you at least pretended to test
stuff before submitting it).
It is a frequent accusation of yours that I don't test things, which is
incorrect. Defending against that makes me a liar twice in your eyes. If
you look more closely at what happens you'll understand that your own
rigid expectations are what causes these problems.
If you thought at all you'd realise that nobody would be stupid enough
to try to sneak untested code into Postgres; all bugs would point
directly back to anybody attempting that. That isn't true just of
Postgres, its true of any group of people working together on any task,
not just software or open source software.
As Greg mentions on another thread, not all patches are *intended* to be
production quality by their authors. Many patches are shared for the
purpose of eliciting general feedback. You yourself encourage a group
development approach and specifically punish those people dropping
completely "finished" code into the queue and expecting it to be
committed as-is. So people produce patches in various states of
readiness, knowing that they may have to produce many versions before it
is finally accepted. Grabbing at a piece of code, then shouting
"unclean, unclean" just destroys the feedback process and leaves
teamwork in tatters.
My arse doesn't need wiping, thanks, nor does my bottom need smacking,
nor are you ever likely to catch me telling fibs. If you think so,
you're wrong and you should reset.
What you will find from me and others, in the past and realistically in
the future too, are patches that vary according to how near to
completion they are. Not the same thing as "completed, yet varying in
quality". If they are incomplete it is because of the idea to receive
feedback at various points. Some patches need almost none e.g. truncate
triggers (1-2 versions), some patches need almost constant feedback e.g.
async commit (24+ versions before commit). The existence of an
intermediate patch in no way signals laziness, lack of intention to
complete or any other failure to appreciate the software development
process.
If you want people to work on Postgres alongside you, I'd appreciate a
software development process that didn't roughly equate to charging at a
machine gun trench across a minefield. If you insist on following that
you should at least stop wondering why it is that the few people to have
made more than a few steps are determined and grim individuals and start
thinking about the many skilled people who have chosen non-combatant
status, and why.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
