tsvector field length limitation

Started by Jonathan Marksalmost 8 years ago4 messagesgeneral
Jump to latest
#1Jonathan Marks
jonathanaverymarks@gmail.com

Hi folks —

We utilize Postgres’ full text search system pretty heavily in our team’s operations and often index tens of millions of records with varying lengths of text. In most cases, the text we need to index is pretty short (no more than. hundreds of words) but in rare cases a single record is very very long (high hundreds of thousands of words or longer). With those records, we run into the max tsvector length requirement "The length of a tsvector (lexemes + positions) must be less than 1 megabyte”

I understand the performance implications of having very long tsvectors (our gin index updates are pretty terrible in some cases) but would really appreciate it if the max tsvector length were larger (even 5MB would make a huge difference) or if that error were a stern warning rather than a hard error.

Is there any way to disable or increase that limit in Postgres 10.3? Perhaps in a future version?

Thank you!
Jonathan

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jonathan Marks (#1)
Re: tsvector field length limitation

Jonathan Marks <jonathanaverymarks@gmail.com> writes:

... we run into the max tsvector length requirement "The length of a tsvector (lexemes + positions) must be less than 1 megabyte”

Is there any way to disable or increase that limit in Postgres 10.3?

No; it's forced by the representation used for tsvector, which stores
lexeme offsets in 20-bit fields (cf WordEntry in
src/include/tsearch/ts_type.h). Perhaps that was short-sighted but
I don't foresee it changing anytime soon. You'd more or less need
a whole new datatype ("bigtsvector"?) to make it happen.

regards, tom lane

#3Jonathan Marks
jonathanaverymarks@gmail.com
In reply to: Tom Lane (#2)
Re: tsvector field length limitation

What if we just didn’t use positional arguments at all? I.e. we just populate the tsvector with lexemes only?

Show quoted text

On Jun 20, 2018, at 10:49 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Jonathan Marks <jonathanaverymarks@gmail.com> writes:

... we run into the max tsvector length requirement "The length of a tsvector (lexemes + positions) must be less than 1 megabyte”

Is there any way to disable or increase that limit in Postgres 10.3?

No; it's forced by the representation used for tsvector, which stores
lexeme offsets in 20-bit fields (cf WordEntry in
src/include/tsearch/ts_type.h). Perhaps that was short-sighted but
I don't foresee it changing anytime soon. You'd more or less need
a whole new datatype ("bigtsvector"?) to make it happen.

regards, tom lane

#4AJG
ayden@gera.co.nz
In reply to: Jonathan Marks (#3)
Re: tsvector field length limitation