Clarification of the "simple" dictionary
Hi. It's not clear to me if the "simple" dictionary uses stopwords or
not, does it?
Can someone please post a complete description of what the "simple"
dict. does?
--
Andreas Joseph Krogh<andreak@officenet.no>
Senior Software Developer / CTO
------------------------+---------------------------------------------+
OfficeNet AS | The most difficult thing in the world is to |
Rosenholmveien 25 | know how to do a thing and to watch |
1414 Troll�sen | somebody else doing it wrong, without |
NORWAY | comment. |
| |
Tlf: +47 24 15 38 90 | |
Fax: +47 24 15 38 91 | |
Mobile: +47 909 56 963 | |
------------------------+---------------------------------------------+
The easiest way to look at this is to give the simple dictionary a
document with to_tsvector() and see if stopwords pop out.
In my experience they do. In my experience, the simple dictionary
just breaks the document down into the space etc. separated words in
the document. It doesn't analyze further.
John
On Jul 22, 2010, at 4:15 PM, Andreas Joseph Krogh wrote:
Show quoted text
Hi. It's not clear to me if the "simple" dictionary uses stopwords
or not, does it?
Can someone please post a complete description of what the "simple"
dict. does?-- Andreas Joseph Krogh<andreak@officenet.no> Senior Software Developer / CTO ------------------------ +---------------------------------------------+ OfficeNet AS | The most difficult thing in the world is to | Rosenholmveien 25 | know how to do a thing and to watch | 1414 Trollåsen | somebody else doing it wrong, without | NORWAY | comment. | | | Tlf: +47 24 15 38 90 | | Fax: +47 24 15 38 91 | | Mobile: +47 909 56 963 | | ------------------------ +---------------------------------------------+--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 07/22/2010 06:27 PM, John Gage wrote:
The easiest way to look at this is to give the simple dictionary a
document with to_tsvector() and see if stopwords pop out.In my experience they do. In my experience, the simple dictionary
just breaks the document down into the space etc. separated words in
the document. It doesn't analyze further.
That's my experience too, I just want to make sure it doesn't actually
have any stopwords which I've missed. Trying many phrases and checking
for stopwords isn't really proving anything.
Can anybody confirm the "simple" dict. only lowercases the words and
"uniques" them?
--
Andreas Joseph Krogh<andreak@officenet.no>
Senior Software Developer / CTO
------------------------+---------------------------------------------+
OfficeNet AS | The most difficult thing in the world is to |
Rosenholmveien 25 | know how to do a thing and to watch |
1414 Troll�sen | somebody else doing it wrong, without |
NORWAY | comment. |
| |
Tlf: +47 24 15 38 90 | |
Fax: +47 24 15 38 91 | |
Mobile: +47 909 56 963 | |
------------------------+---------------------------------------------+
Don't guess, but read docs
http://www.postgresql.org/docs/8.4/interactive/textsearch-dictionaries.html#TEXTSEARCH-SIMPLE-DICTIONARY
12.6.2. Simple Dictionary
The simple dictionary template operates by converting the input token to lower case and checking it against a file of stop words. If it is found in the file then an empty array is returned, causing the token to be discarded. If not, the lower-cased form of the word is returned as the normalized lexeme. Alternatively, the dictionary can be configured to report non-stop-words as unrecognized, allowing them to be passed on to the next dictionary in the list.
d=# \dFd+ simple
List of text search dictionaries
Schema | Name | Template | Init options | Description
------------+--------+-------------------+--------------+-----------------------------------------------------------
pg_catalog | simple | pg_catalog.simple | | simple dictionary: just lower case and check for stopword
By default it has no Init options, so it doesn't check for stopwords.
On Thu, 22 Jul 2010, Andreas Joseph Krogh wrote:
On 07/22/2010 06:27 PM, John Gage wrote:
The easiest way to look at this is to give the simple dictionary a document
with to_tsvector() and see if stopwords pop out.In my experience they do. In my experience, the simple dictionary just
breaks the document down into the space etc. separated words in the
document. It doesn't analyze further.That's my experience too, I just want to make sure it doesn't actually have
any stopwords which I've missed. Trying many phrases and checking for
stopwords isn't really proving anything.Can anybody confirm the "simple" dict. only lowercases the words and
"uniques" them?
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
On 07/22/2010 07:44 PM, Oleg Bartunov wrote:
Don't guess, but read docs
http://www.postgresql.org/docs/8.4/interactive/textsearch-dictionaries.html#TEXTSEARCH-SIMPLE-DICTIONARY12.6.2. Simple Dictionary
The simple dictionary template operates by converting the input token
to lower case and checking it against a file of stop words. If it is
found in the file then an empty array is returned, causing the token
to be discarded. If not, the lower-cased form of the word is returned
as the normalized lexeme. Alternatively, the dictionary can be
configured to report non-stop-words as unrecognized, allowing them to
be passed on to the next dictionary in the list.d=# \dFd+ simple
List of text search
dictionaries
Schema | Name | Template | Init options
| Description
------------+--------+-------------------+--------------+-----------------------------------------------------------pg_catalog | simple | pg_catalog.simple | | simple
dictionary: just lower case and check for stopwordBy default it has no Init options, so it doesn't check for stopwords.
Guess what - I *have* read the docs which sais "...and checking it
against a file of stop words". What was unclear to me was whether or not
it was configured with a stopwords-file or not as default, which is not
the case I understand from your reply. Very good, fits my needs like a
glove:-) It might be worth considering updating the docs to make this
clearer?
So - can we rely on "simple" to remain this way forever (no Init
options) or is it better to make a copy of it with the same properties
as today?
It seems "simple" + the unaccent dict. available in 9.0 saves my day,
thanks Mr. Bartunov.
--
Andreas Joseph Krogh<andreak@officenet.no>
Senior Software Developer / CTO
------------------------+---------------------------------------------+
OfficeNet AS | The most difficult thing in the world is to |
Rosenholmveien 25 | know how to do a thing and to watch |
1414 Troll�sen | somebody else doing it wrong, without |
NORWAY | comment. |
| |
Tlf: +47 24 15 38 90 | |
Fax: +47 24 15 38 91 | |
Mobile: +47 909 56 963 | |
------------------------+---------------------------------------------+
Andreas,
I'd create myself copy of dictionary to be independent on system changes.
Oleg
On Thu, 22 Jul 2010, Andreas Joseph Krogh wrote:
On 07/22/2010 07:44 PM, Oleg Bartunov wrote:
Don't guess, but read docs
http://www.postgresql.org/docs/8.4/interactive/textsearch-dictionaries.html#TEXTSEARCH-SIMPLE-DICTIONARY12.6.2. Simple Dictionary
The simple dictionary template operates by converting the input token to
lower case and checking it against a file of stop words. If it is found in
the file then an empty array is returned, causing the token to be
discarded. If not, the lower-cased form of the word is returned as the
normalized lexeme. Alternatively, the dictionary can be configured to
report non-stop-words as unrecognized, allowing them to be passed on to the
next dictionary in the list.d=# \dFd+ simple
List of text search dictionaries
Schema | Name | Template | Init options |
Description
------------+--------+-------------------+--------------+-----------------------------------------------------------
pg_catalog | simple | pg_catalog.simple | | simple
dictionary: just lower case and check for stopwordBy default it has no Init options, so it doesn't check for stopwords.
Guess what - I *have* read the docs which sais "...and checking it against a
file of stop words". What was unclear to me was whether or not it was
configured with a stopwords-file or not as default, which is not the case I
understand from your reply. Very good, fits my needs like a glove:-) It might
be worth considering updating the docs to make this clearer?So - can we rely on "simple" to remain this way forever (no Init options) or
is it better to make a copy of it with the same properties as today?It seems "simple" + the unaccent dict. available in 9.0 saves my day, thanks
Mr. Bartunov.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
By default it has no Init options, so it doesn't check for stopwords.
In the first place, this functionality is a rip-snorting home run on
Postgres. I congratulate Oleg who I believe is one of the authors.
In the second, I too had not read (carefully) the documentation and am
very happy to find that I can eliminate stop words with 'simple'.
That will be a tremendous convenience going forward.
It turns out that using 'english' and getting stemmed lexemes is
extremely convenient too, but this functionality in 'simple' is
excellent.
Thanks,
John