Use of "token" vs "lexeme" in text search documentation
The current documentation seems a bit inconsistent in its use of the
terms "token" and "lexeme". The majority of the text seems to use
"lexeme" exclusively, which is inconsistent with the fact that the
term "token" is exposed by ts_token_type() and friends. But there
are a few places that seem to use "lexeme" to mean something returned
by a dictionary.
I was considering trying to adopt these conventions:
* What a parser returns is a "token".
* When a dictionary recognizes a token, what it returns is a "lexeme".
This would make the phrase "normalized lexeme" redundant, since we
don't call it a lexeme at all unless it's been normalized.
Comments?
regards, tom lane
On Mon, 15 Oct 2007, Tom Lane wrote:
The current documentation seems a bit inconsistent in its use of the
terms "token" and "lexeme". The majority of the text seems to use
"lexeme" exclusively, which is inconsistent with the fact that the
term "token" is exposed by ts_token_type() and friends. But there
are a few places that seem to use "lexeme" to mean something returned
by a dictionary.I was considering trying to adopt these conventions:
* What a parser returns is a "token".
* When a dictionary recognizes a token, what it returns is a "lexeme".
This would make the phrase "normalized lexeme" redundant, since we
don't call it a lexeme at all unless it's been normalized.Comments?
Hmm, you say what I always thought. I'd be happy if you stress this in
docs.
regards, tom lane
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83