BUG #7793: tsearch_data thesaurus size limit

davios@gmail.com writes:

[ thesaurus dictionary fails for more than 64K entries ]

I see a whole bunch of uses of "uint16" in
src/backend/tsearch/dict_thesaurus.c. It's not immediately clear which
of these would need to be widened to support more entries, or what the
storage cost of doing that would be. We probably should at least put in
a range check so that you get a clean failure instead of a crash though.

regards, tom lane

--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Ok thanks for your reply Tom.

We have made on our side some update on this file
src/backend/tsearch/dict_thesaurus.c
Then we recompile PG 9.2.2 with this patch and now the thesaurus works fine
with more than 64k entries and queries runtime is always low as expected.

Here is our update of the file:

 typedef struct LexemeInfo
 {
- *uint16* idsubst; /* entry's number in DictThesaurus->subst */
+ *uint32* idsubst; /* entry's number in DictThesaurus->subst */
  uint16 posinsubst; /* pos info in entry */

...

 static void
-newLexeme(DictThesaurus *d, char *b, char *e, *uint16* idsubst, uint16
posinsubst)
+newLexeme(DictThesaurus *d, char *b, char *e, *uint32* idsubst, uint16
posinsubst)
 {
  TheLexeme  *ptr;

...

 static void
-addWrd(DictThesaurus *d, char *b, char *e, *uint16* idsubst, uint16 nwrd,
uint16 posinsubst, bool useasis)
+addWrd(DictThesaurus *d, char *b, char *e, *uint32* idsubst, uint16 nwrd,
uint16 posinsubst, bool useasis)
 {

...

thesaurusRead(char *filename, DictThesaurus *d) {
tsearch_readline_state trst;
- *uint16* idsubst = 0;
+ *uint32* idsubst = 0;
bool useasis = false;

...

 static bool
-matchIdSubst(LexemeInfo *stored, *uint16* idsubst)
+matchIdSubst(LexemeInfo *stored, *uint32* idsubst)
 {
  bool res = true;

Kind regards.
David

On Mon, Jan 7, 2013 at 1:41 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: