tsearch2 problems / limitations

On Tue, 22 Feb 2005, James Croft wrote:

Hi,

I've successfully added tsearch2 to an existing database and the speedup of
searches is brilliant. I'm now trying to extend this to other parts of our
system.

One of the tables holds reasonable amounts of text, some fields hold up to
2Mb. When I try and run

UPDATE table SET idxfti=to_tsvector('default', field);

it runs for a while then aborts with the following message

ERROR: value is too big

Does anyone know what the problem might be here? Does tsearch2 have
configurable (or compilable) settings that could get around this?

this is tsearch2 limitation on length of ts_vector about 1mb.
you may stop using postional information for this vector.

Many Thanks,
James

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

On Tue, Feb 22, 2005 at 01:29:45PM +0000, James Croft wrote:

One of the tables holds reasonable amounts of text, some fields hold up
to 2Mb. When I try and run

UPDATE table SET idxfti=to_tsvector('default', field);

it runs for a while then aborts with the following message

ERROR: value is too big

README.tsearch2 says that the maximum size of a tsvector is 2^20
bytes (1M). You can see that limit enforced in the makevalue()
function in tsvector.c (similarly in pushquery() in query.c):

if (cur - str > MAXSTRPOS)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("value is too big")));

MAXSTRPOS is defined in tsvector.h:

#define MAXSTRPOS ( 1<<20 )

Maybe Oleg will reply and say whether it's safe to change that
or not.

--
Michael Fuhr
http://www.fuhr.org/~mfuhr/