snowball ASCII stemmer configuration

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

There are two cases where these two columns are not the same:

hindi english \
russian english \

The second one is old; the first one I added using the second one as
example. But I wonder what the rationale for this is. Maybe for hindi
one could make some kind of cultural argument, but for russian this
seems entirely arbitrary.

Perhaps it is, but we have actual Russians who think it's a good idea.
I recall questioning that point some years ago, and Oleg replied that
they'd done that intentionally because (a) technical Russian uses a lot
of English words, and (b) it's easy to tell which is which thanks to
the disjoint letter sets.

Whether the same is true for Hindi, I have no idea.

Moreover, AFAIK, the following other languages do not use Latin-based
alphabets:

arabic arabic \
greek greek \
nepali nepali \
tamil tamil \

Hmm. I think all of those entries are ones that got added by me while
absorbing post-2007 Snowball updates, and I confess that I did not think
about this point. Maybe these should be changed.

regards, tom lane

On Tue, Jun 16, 2020 at 4:53 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

There are two cases where these two columns are not the same:

hindi english \
russian english \

The second one is old; the first one I added using the second one as
example. But I wonder what the rationale for this is. Maybe for hindi
one could make some kind of cultural argument, but for russian this
seems entirely arbitrary.

Perhaps it is, but we have actual Russians who think it's a good idea.
I recall questioning that point some years ago, and Oleg replied that
they'd done that intentionally because (a) technical Russian uses a lot
of English words, and (b) it's easy to tell which is which thanks to
the disjoint letter sets.

Yes, you are right.

Whether the same is true for Hindi, I have no idea.

Moreover, AFAIK, the following other languages do not use Latin-based
alphabets:

arabic arabic \
greek greek \
nepali nepali \
tamil tamil \

Hmm. I think all of those entries are ones that got added by me while
absorbing post-2007 Snowball updates, and I confess that I did not think
about this point. Maybe these should be changed.

regards, tom lane

--
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

I wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Moreover, AFAIK, the following other languages do not use Latin-based
alphabets:

arabic arabic \
greek greek \
nepali nepali \
tamil tamil \

Hmm. I think all of those entries are ones that got added by me while
absorbing post-2007 Snowball updates, and I confess that I did not think
about this point. Maybe these should be changed.

After further reflection, I think these are indeed mistakes and we should
change them all. The argument for the Russian/English case, AIUI, is
"if we come across an all-ASCII word, it is most certainly not Russian,
and the most likely Latin-based language is English". Given the world
as it is, I think the same argument works for all non-Latin-alphabet
languages. Obviously specific applications might have a different idea
of the best fallback language, but that's why we let users make their
own text search configurations. For general-purpose use, falling back
to English seems reasonable. And we can be dead certain that applying
a Greek stemmer to an ASCII word will do nothing useful, so the
configuration choice shown above is unhelpful.

regards, tom lane

On Jun 16, 2020, at 7:37 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I wrote:

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Moreover, AFAIK, the following other languages do not use Latin-based
alphabets:

arabic arabic \
greek greek \
nepali nepali \
tamil tamil \

Hmm. I think all of those entries are ones that got added by me while
absorbing post-2007 Snowball updates, and I confess that I did not think
about this point. Maybe these should be changed.

After further reflection, I think these are indeed mistakes and we should
change them all. The argument for the Russian/English case, AIUI, is
"if we come across an all-ASCII word, it is most certainly not Russian,
and the most likely Latin-based language is English". Given the world
as it is, I think the same argument works for all non-Latin-alphabet
languages. Obviously specific applications might have a different idea
of the best fallback language, but that's why we let users make their
own text search configurations. For general-purpose use, falling back
to English seems reasonable. And we can be dead certain that applying
a Greek stemmer to an ASCII word will do nothing useful, so the
configuration choice shown above is unhelpful.

I am a bit surprised to see that you are right about this, because non-latin languages often have transliteration/romanization schemes for writing the language in the Latin alphabet, developed before computers had wide spread adoption of non-ASCII character sets, and still in use today for text messaging. I expected to find stemming rules for transliterated words, but can't find any indication of that, neither in the postgres sources, nor in the snowball sources I pulled from their repo. Is there some architectural separation of stemming from transliteration such that we'd never need to worry about it? If snowball ever published stemmers for transliterated text, we might have to revisit this issue, but for now your proposed change sounds fine to me.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Mark Dilger <mark.dilger@enterprisedb.com> writes:

I am a bit surprised to see that you are right about this, because non-latin languages often have transliteration/romanization schemes for writing the language in the Latin alphabet, developed before computers had wide spread adoption of non-ASCII character sets, and still in use today for text messaging. I expected to find stemming rules for transliterated words, but can't find any indication of that, neither in the postgres sources, nor in the snowball sources I pulled from their repo. Is there some architectural separation of stemming from transliteration such that we'd never need to worry about it? If snowball ever published stemmers for transliterated text, we might have to revisit this issue, but for now your proposed change sounds fine to me.

Agreed, if the Snowball stemmers worked on romanized texts then the
situation would be different. But they don't, AFAICS. Don't know
if that is architectural, or a policy decision, or just lack of
round tuits.

The thing that I actually find a bit shaky in this area is our
architectural decision to route words to different dictionaries
depending on whether they are all-ASCII or not. AIUI that was
done purely on the basis of the Russian/English case; it would
fail badly if say you wanted to separate Russian from French.
However, I have no great desire to revisit that design right now.

regards, tom lane

On 2020-06-16 16:37, Tom Lane wrote:

After further reflection, I think these are indeed mistakes and we should
change them all. The argument for the Russian/English case, AIUI, is
"if we come across an all-ASCII word, it is most certainly not Russian,
and the most likely Latin-based language is English". Given the world
as it is, I think the same argument works for all non-Latin-alphabet
languages. Obviously specific applications might have a different idea
of the best fallback language, but that's why we let users make their
own text search configurations. For general-purpose use, falling back
to English seems reasonable. And we can be dead certain that applying
a Greek stemmer to an ASCII word will do nothing useful, so the
configuration choice shown above is unhelpful.

Do we *have* to have an ASCII stemmer that corresponds to an actual
language? Couldn't we use the simple stemmer or no stemmer at all?

In my experience, ASCII text in, say, Russian or Greek will typically be
acronyms or brand names or the like, and there doesn't seem to be a
great need to stem that kind of thing. Just doing nothing seems at
least as good.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:

Do we *have* to have an ASCII stemmer that corresponds to an actual
language? Couldn't we use the simple stemmer or no stemmer at all?
In my experience, ASCII text in, say, Russian or Greek will typically be
acronyms or brand names or the like, and there doesn't seem to be a
great need to stem that kind of thing. Just doing nothing seems at
least as good.

Well, I have no horse in this race. But the reason it's like this for
Russian is that Oleg, Teodor, and crew set it up that way ages ago.
I'd tend to defer to their opinion about what's the most usable
configuration for Russian. You could certainly argue that the situation
is different for $other-language ... but without some hard evidence for
that position, making these cases all behave similarly seems like a
reasonable approach.

regards, tom lane