Backend crash with tsearch

"Magnus Naeslund(f)" <mag@fbab.net> writes:

The backend dies in a segfault.

Backtrace:

#0 0x02d10000 in ?? ()
#1 0x401faf48 in ?? ()
#2 0x401fb5e6 in ?? ()
#3 0x080d8f5c in ExecMakeFunctionResult (fcache=0x82d3710,
arguments=0x82ce170, econtext=0x82d3580, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839

Did you compile tsearch with debug support? If so, the lack of any
symbolic info here must mean that gdb didn't know where to find the
tsearch .so module. You could get a more useful trace by telling
gdb
sharedlibrary /path/to/tsearch.so

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> wrote:

"Magnus Naeslund(f)" <mag@fbab.net> writes:

The backend dies in a segfault.

Backtrace:

#0 0x02d10000 in ?? ()
#1 0x401faf48 in ?? ()
#2 0x401fb5e6 in ?? ()
#3 0x080d8f5c in ExecMakeFunctionResult (fcache=0x82d3710,
arguments=0x82ce170, econtext=0x82d3580, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839

Did you compile tsearch with debug support? If so, the lack of any
symbolic info here must mean that gdb didn't know where to find the
tsearch .so module. You could get a more useful trace by telling
gdb
sharedlibrary /path/to/tsearch.so

regards, tom lane

I'm working on it (--enable-debug --enable-cassert).
It's either that it can't load the lib (shouldn't it complain?) or it's
a bad pointer.
We'll find out, i hope...

Magnus

"Magnus Naeslund\(f\)" <mag@fbab.net> writes:

It's either that it can't load the lib (shouldn't it complain?) or it's
a bad pointer.

Be sure to eliminate the possibility that you're loading the wrong
version of the .so (ie, loading a 7.2 tsearch.so into 7.3). People
get bit by that quite frequently right after an upgrade ...

regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> wrote:

Be sure to eliminate the possibility that you're loading the wrong
version of the .so (ie, loading a 7.2 tsearch.so into 7.3). People
get bit by that quite frequently right after an upgrade ...

Well, this is a clean install, so that isn't a problem.

Magnus

On Tue, 3 Dec 2002, Magnus Naeslund(f) wrote:

I'm evaluating tsearch contrib module, and i get a backend crash when
i'm about to use a tsearch function.

When i issue
update things set nidx=txt2txtidx(productname),
didx=txt2txtidx(longdescription);

The backend dies in a segfault.
The system is redhat 7.3 dual athlon w/ 1GB memory.
Postgresql is compiled with -march=athlon -O3.
initdb -E LATIN1

Please, tell us postgresql version. Did you reinstall tsearch after
upgrading ? Test-suite (data, sql) demonstrated the problem would be
nice.

I have a huge shared buffer count (65536).

I'll reinstall tsearch and try again soon.
Is it necesary to install OpenFTS contrib aswell, or do i get away with
only installing tsearch?

For contrib/tsearch you need only tsearch :)

Now i do both...

Backtrace:

#0 0x02d10000 in ?? ()
#1 0x401faf48 in ?? ()
#2 0x401fb5e6 in ?? ()
#3 0x080d8f5c in ExecMakeFunctionResult (fcache=0x82d3710,
arguments=0x82ce170, econtext=0x82d3580, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839
#4 0x080d99a3 in ExecEvalExpr (expression=0x82ce188,
econtext=0x82d3580, isNull=0xbfffec8f "", isDone=0xbfffecd8)
at execQual.c:1168
#5 0x080d9d44 in ExecTargetList (targetlist=0x82ce3d8, nodomains=21,
targettype=0x82cf230, values=0x82d4488,
econtext=0x82d3580, isDone=0xbfffee78) at execQual.c:2058
#6 0x080da13f in ExecProject (projInfo=0x82d3b08, isDone=0xbfffee78) at
execQual.c:2282
#7 0x080da229 in ExecScan (node=0x82cfeb8, accessMtd=0x80e1270
<SeqNext>) at execScan.c:133
#8 0x080e1093 in ExecSeqScan (node=0x82cfeb8) at nodeSeqscan.c:133
#9 0x080d7d9c in ExecProcNode (node=0x82cfeb8, parent=0x0) at
execProcnode.c:291
#10 0x080d6a47 in ExecutePlan (estate=0x82d0000, plan=0x82cfeb8,
operation=CMD_UPDATE, numberTuples=0,
direction=ForwardScanDirection, destfunc=0x82d3b30) at
execMain.c:954
#11 0x080d7682 in ExecutorRun (queryDesc=0x82d35f0, estate=0x82d0000,
direction=ForwardScanDirection, count=0) at execMain.c:195
#12 0x0812a8cb in ProcessQuery (parsetree=0x82cb1c8, plan=0x82cfeb8,
dest=Remote, completionTag=0xbffff060 "") at pquery.c:242
#13 0x08128b81 in pg_exec_query_string (query_string=0x82cb0a8,
dest=Remote, parse_context=0x8291cd0) at postgres.c:838
#14 0x08129b50 in PostgresMain (argc=4, argv=0xbffff2e0,
username=0x827ccd1 "mag") at postgres.c:2016
#15 0x0810f0c4 in DoBackend (port=0x827cba0) at postmaster.c:2293
#16 0x0810e9dc in BackendStartup (port=0x827cba0) at postmaster.c:1915
#17 0x0810de8d in ServerLoop () at postmaster.c:1000
#18 0x0810da24 in PostmasterMain (argc=1, argv=0x8245640) at
postmaster.c:779
#19 0x080ea5c2 in main (argc=1, argv=0xbffffc74) at main.c:210
#20 0x42017589 in __libc_start_main () from /lib/i686/libc.so.6

Magnus

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov <oleg@sai.msu.su> wrote:

Please, tell us postgresql version. Did you reinstall tsearch after
upgrading ? Test-suite (data, sql) demonstrated the problem would be
nice.

pgsql 7.3, about 700mb text database with product descriptions.
I'm working on isolation the behavior, the tsearch make installcheck
seems to be crashing aswell.
Is a "psql regression < tsearch.sql" needed, or is that done
automatically in the installcheck?

For contrib/tsearch you need only tsearch :)

:)

I hope this is because of something silly.

Magnus

More info, the gdb> sharedlibrary loaded some more symbols:

(gdb) bt
#0 0x02d10000 in ?? ()
#1 0x401faf48 in parsetext (prs=0xbfffea60, buf=0x4277eb3c "Can - Live
1971-77", buflen=18) at txtidx.c:366
#2 0x401fb5e6 in txt2txtidx (fcinfo=0xbfffeaf0) at txtidx.c:487
#3 0x080ec45c in ExecMakeFunctionResult (fcache=0x83172bc,
arguments=0x831187c, econtext=0x8317114, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839
#4 0x080ed023 in ExecEvalExpr (expression=0x8311898,
econtext=0x8317114, isNull=0xbfffec8f "", isDone=0xbfffecd8)
at execQual.c:1168
#5 0x080ed3c4 in ExecTargetList (targetlist=0x8311b20, nodomains=21,
targettype=0x8312b1c, values=0x83180a0,
econtext=0x8317114, isDone=0xbfffee78) at execQual.c:2058
#6 0x080ed7bf in ExecProject (projInfo=0x8317f90, isDone=0xbfffee78) at
execQual.c:2282
#7 0x080ed8a9 in ExecScan (node=0x8315e60, accessMtd=0x80f4fa0
<SeqNext>) at execScan.c:133
#8 0x080f4e73 in ExecSeqScan (node=0x8315e60) at nodeSeqscan.c:133
#9 0x080eafbc in ExecProcNode (node=0x8315e60, parent=0x0) at
execProcnode.c:291
#10 0x080e99f7 in ExecutePlan (estate=0x83161ac, plan=0x8315e60,
operation=CMD_UPDATE, numberTuples=0,
direction=ForwardScanDirection, destfunc=0x8317fbc) at
execMain.c:954
#11 0x080ea999 in ExecutorRun (queryDesc=0x831718c, estate=0x83161ac,
direction=ForwardScanDirection, count=0) at execMain.c:195
#12 0x08143b9b in ProcessQuery (parsetree=0x830c8c4, plan=0x8315e60,
dest=Remote, completionTag=0xbffff060 "") at pquery.c:242
#13 0x08141dc1 in pg_exec_query_string (query_string=0x830c79c,
dest=Remote, parse_context=0x82d6e88) at postgres.c:838
#14 0x08142e1d in PostgresMain (argc=4, argv=0xbffff2e0,
username=0x82c23a9 "mag") at postgres.c:2016
#15 0x08125544 in DoBackend (port=0x82c2278) at postmaster.c:2293
#16 0x08124e5c in BackendStartup (port=0x82c2278) at postmaster.c:1915
#17 0x0812430d in ServerLoop () at postmaster.c:1000
#18 0x08123e94 in PostmasterMain (argc=1, argv=0x8276d00) at
postmaster.c:779
#19 0x080fefe2 in main (argc=1, argv=0xbffffc74) at main.c:210
#20 0x42017589 in __libc_start_main () from /lib/i686/libc.so.6

Magnus

On Tue, 3 Dec 2002, Magnus Naeslund(f) wrote:

Oleg Bartunov <oleg@sai.msu.su> wrote:

Please, tell us postgresql version. Did you reinstall tsearch after
upgrading ? Test-suite (data, sql) demonstrated the problem would be
nice.

pgsql 7.3, about 700mb text database with product descriptions.
I'm working on isolation the behavior, the tsearch make installcheck
seems to be crashing aswell.
Is a "psql regression < tsearch.sql" needed, or is that done
automatically in the installcheck?

Magnus,

what is an output of 'make installcheck' ?

For contrib/tsearch you need only tsearch :)

:)

I hope this is because of something silly.

Magnus

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Does it crashed?
# select txt2txtidx('Can - Live 1971-77');

Line txtidx.c:366 :
lemm = lemmatize(token, &lenlemm, type);

lemmatize() is defined in morph.c. Did you use another modules for postgresql?

It seems to me that we see a name conflict. Function lemmatize is defined in
somewhere also.

Magnus Naeslund(f) wrote:

More info, the gdb> sharedlibrary loaded some more symbols:

(gdb) bt
#0 0x02d10000 in ?? ()
#1 0x401faf48 in parsetext (prs=0xbfffea60, buf=0x4277eb3c "Can - Live
1971-77", buflen=18) at txtidx.c:366
#2 0x401fb5e6 in txt2txtidx (fcinfo=0xbfffeaf0) at txtidx.c:487
#3 0x080ec45c in ExecMakeFunctionResult (fcache=0x83172bc,
arguments=0x831187c, econtext=0x8317114, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839
#4 0x080ed023 in ExecEvalExpr (expression=0x8311898,
econtext=0x8317114, isNull=0xbfffec8f "", isDone=0xbfffecd8)
at execQual.c:1168
#5 0x080ed3c4 in ExecTargetList (targetlist=0x8311b20, nodomains=21,
targettype=0x8312b1c, values=0x83180a0,
econtext=0x8317114, isDone=0xbfffee78) at execQual.c:2058
#6 0x080ed7bf in ExecProject (projInfo=0x8317f90, isDone=0xbfffee78) at
execQual.c:2282
#7 0x080ed8a9 in ExecScan (node=0x8315e60, accessMtd=0x80f4fa0
<SeqNext>) at execScan.c:133
#8 0x080f4e73 in ExecSeqScan (node=0x8315e60) at nodeSeqscan.c:133
#9 0x080eafbc in ExecProcNode (node=0x8315e60, parent=0x0) at
execProcnode.c:291
#10 0x080e99f7 in ExecutePlan (estate=0x83161ac, plan=0x8315e60,
operation=CMD_UPDATE, numberTuples=0,
direction=ForwardScanDirection, destfunc=0x8317fbc) at
execMain.c:954
#11 0x080ea999 in ExecutorRun (queryDesc=0x831718c, estate=0x83161ac,
direction=ForwardScanDirection, count=0) at execMain.c:195
#12 0x08143b9b in ProcessQuery (parsetree=0x830c8c4, plan=0x8315e60,
dest=Remote, completionTag=0xbffff060 "") at pquery.c:242
#13 0x08141dc1 in pg_exec_query_string (query_string=0x830c79c,
dest=Remote, parse_context=0x82d6e88) at postgres.c:838
#14 0x08142e1d in PostgresMain (argc=4, argv=0xbffff2e0,
username=0x82c23a9 "mag") at postgres.c:2016
#15 0x08125544 in DoBackend (port=0x82c2278) at postmaster.c:2293
#16 0x08124e5c in BackendStartup (port=0x82c2278) at postmaster.c:1915
#17 0x0812430d in ServerLoop () at postmaster.c:1000
#18 0x08123e94 in PostmasterMain (argc=1, argv=0x8276d00) at
postmaster.c:779
#19 0x080fefe2 in main (argc=1, argv=0xbffffc74) at main.c:210
#20 0x42017589 in __libc_start_main () from /lib/i686/libc.so.6

Magnus

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

--
Teodor Sigaev
teodor@stack.net

Oleg Bartunov <oleg@sai.msu.su> wrote:

Magnus,

what is an output of 'make installcheck' ?

As i said, it segfaults:

[root@fet1b tsearch]# gmake installcheck
gmake -C ../../src/test/regress pg_regress
gmake[1]: Entering directory `/usr/src/postgresql-7.3/src/test/regress'
gmake[1]: `pg_regress' is up to date.
gmake[1]: Leaving directory `/usr/src/postgresql-7.3/src/test/regress'
../../src/test/regress/pg_regress tsearch
(using postmaster on Unix socket, default port)
============== dropping database "regression" ==============
DROP DATABASE
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
============== dropping regression test user accounts ==============
============== installing PL/pgSQL ==============
============== running regression test queries ==============
test tsearch ... FAILED

======================
1 of 1 tests failed.
======================

The differences that caused some tests to fail can be viewed in the
file `./regression.diffs'. A copy of the test summary that you see
above is saved in the file `./regression.out'.

gmake: *** [installcheck] Error 1

The regression.diff is attached.

Magnus

Teodor Sigaev <teodor@stack.net> wrote:

Does it crashed?
# select txt2txtidx('Can - Live 1971-77');

Line txtidx.c:366 :
lemm = lemmatize(token, &lenlemm, type);

lemmatize() is defined in morph.c. Did you use another modules for
postgresql?

It seems to me that we see a name conflict. Function lemmatize is
defined in somewhere also.

This is what i found out aswell, but isn't lemmatize resolved at
compiletime?
The only other module is Search-OpenFTS.
I'll check if i see any conflicts in $prefix/lib

Magnus

Pls, send backtrace... :)

Magnus Naeslund(f) wrote:

Oleg Bartunov <oleg@sai.msu.su> wrote:

Magnus,

what is an output of 'make installcheck' ?

As i said, it segfaults:

[root@fet1b tsearch]# gmake installcheck
gmake -C ../../src/test/regress pg_regress
gmake[1]: Entering directory `/usr/src/postgresql-7.3/src/test/regress'
gmake[1]: `pg_regress' is up to date.
gmake[1]: Leaving directory `/usr/src/postgresql-7.3/src/test/regress'
../../src/test/regress/pg_regress tsearch
(using postmaster on Unix socket, default port)
============== dropping database "regression" ==============
DROP DATABASE
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
============== dropping regression test user accounts ==============
============== installing PL/pgSQL ==============
============== running regression test queries ==============
test tsearch ... FAILED

======================
1 of 1 tests failed.
======================

The differences that caused some tests to fail can be viewed in the
file `./regression.diffs'. A copy of the test summary that you see
above is saved in the file `./regression.out'.

gmake: *** [installcheck] Error 1

The regression.diff is attached.

Magnus

------------------------------------------------------------------------

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

--
Teodor Sigaev
teodor@stack.net

Teodor Sigaev <teodor@stack.net> wrote:

Pls, send backtrace... :)

I already have, twice.

Magnus

Teodor Sigaev wrote:

Does it crashed?
# select txt2txtidx('Can - Live 1971-77');

Line txtidx.c:366 :
lemm = lemmatize(token, &lenlemm, type);

lemmatize() is defined in morph.c. Did you use another modules for
postgresql?

It seems to me that we see a name conflict. Function lemmatize is
defined in somewhere also.

Magnus Naeslund(f) wrote:

More info, the gdb> sharedlibrary loaded some more symbols:

(gdb) bt
#0 0x02d10000 in ?? ()
#1 0x401faf48 in parsetext (prs=0xbfffea60, buf=0x4277eb3c "Can - Live
1971-77", buflen=18) at txtidx.c:366
#2 0x401fb5e6 in txt2txtidx (fcinfo=0xbfffeaf0) at txtidx.c:487
#3 0x080ec45c in ExecMakeFunctionResult (fcache=0x83172bc,
arguments=0x831187c, econtext=0x8317114, isNull=0xbfffec8f "",
isDone=0xbfffecd8) at execQual.c:839

Pointers 0x40* - functions in tsearch.so, 0x08 - postgres file.

So first string '#0 0x02d10000 in ?? ()' has pointer to 'black hole'.....
Very strange....

--
Teodor Sigaev
teodor@stack.net

On Tue, 3 Dec 2002, Magnus Naeslund(f) wrote:

Teodor Sigaev <teodor@stack.net> wrote:

Pls, send backtrace... :)

I already have, twice.

Magnus, we need backtrace from 'make installcheck'

Magnus

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Oleg Bartunov <oleg@sai.msu.su> wrote:

Magnus, we need backtrace from 'make installcheck'

As you wish...
This is a bt taken from a core file this time (the other ones were from
attached processes).
The whole thing has been recompiled with no additional compiler flags
(i.e. removed -march=athlon -O3), but still with --enable-debug
and --enable-cassert.

(gdb) bt
#0 0x02d10000 in ?? ()
#1 0x401f9071 in parsetext (prs=0xbfffeb70,
buf=0x82ca550 "345 qwe@efd.r ' http://www.com/
http://aew.werc.ewr/?ad=qwe&amp;dw 1aew.werc.ewr/?ad=qwe&dw 2aew.werc.ewr
http://3aew.werc.ewr/?ad=qwe&amp;dw http://4aew.werc.ewr
http://5aew.werc.ewr:8100/? ad=qwe&dw 6aew.w"..., buflen=564) at
txtidx.c:366
#2 0x401f93c6 in txt2txtidx (fcinfo=0xbfffebe0) at txtidx.c:487
#3 0x080e2af0 in ExecMakeFunctionResult (fcache=0x82cad44,
arguments=0x82ca958, econtext=0x82cabc0, isNull=0xbfffed3f "",
isDone=0xbfffed40) at execQual.c:839
#4 0x080e2fc1 in ExecEvalFunc (funcClause=0x82ca974,
econtext=0x82cabc0, isNull=0xbfffed3f "", isDone=0xbfffed40)
at execQual.c:1168
#5 0x080e3755 in ExecEvalExpr (expression=0x82ca974,
econtext=0x82cabc0, isNull=0xbfffed3f "", isDone=0xbfffed40)
at execQual.c:1715
#6 0x080e3a34 in ExecTargetList (targetlist=0x82ca9a0, nodomains=1,
targettype=0x82cac0c, values=0x82cacfc, econtext=0x82cabc0,
isDone=0xbfffef28) at execQual.c:2058
#7 0x080e3cc9 in ExecProject (projInfo=0x82cacd0, isDone=0xbfffef28) at
execQual.c:2282
#8 0x080e9dcb in ExecResult (node=0x82ca9bc) at nodeResult.c:160
#9 0x080e17e9 in ExecProcNode (node=0x82ca9bc, parent=0x0) at
execProcnode.c:280
#10 0x080e05c3 in ExecutePlan (estate=0x82caa74, plan=0x82ca9bc,
operation=CMD_SELECT, numberTuples=0,
direction=ForwardScanDirection, destfunc=0x82cad18) at
execMain.c:954
#11 0x080dfbfd in ExecutorRun (queryDesc=0x82caa48, estate=0x82caa74,
direction=ForwardScanDirection, count=0) at execMain.c:195
#12 0x08132e33 in ProcessQuery (parsetree=0x82bb344, plan=0x82ca9bc,
dest=Remote, completionTag=0xbffff0b0 "") at pquery.c:242
#13 0x08131414 in pg_exec_query_string (query_string=0x82ba124,
dest=Remote, parse_context=0x8284684) at postgres.c:838
#14 0x08132535 in PostgresMain (argc=4, argv=0xbffff2e0,
username=0x826fe61 "root") at postgres.c:2016
#15 0x08117b34 in DoBackend (port=0x826fd30) at postmaster.c:2293
#16 0x08117486 in BackendStartup (port=0x826fd30) at postmaster.c:1915
#17 0x08116689 in ServerLoop () at postmaster.c:1000
#18 0x0811623a in PostmasterMain (argc=1, argv=0x82568a0) at
postmaster.c:779
#19 0x080f3293 in main (argc=1, argv=0xbffffc74) at main.c:210
#20 0x42017589 in __libc_start_main () from /lib/i686/libc.so.6

Magnus

Some more (useless) info.

objdump -x /lib/*.so /usr/lib/*.so /lib/i686/*.so /usr/kerberos/lib/*.so
/usr/local/pgsql/bin/* /usr/local/pgsql/lib/*.so | grep lemmatize

reviels only one lemmatize symbol.

The offending address 0x02d10000 is not mapped anywhere in the address
space according to /proc/<pid>/maps.

Nice that the coredump is 522MB ;)

Magnus

As you wish...
This is a bt taken from a core file this time (the other ones were from
attached processes).
The whole thing has been recompiled with no additional compiler flags
(i.e. removed -march=athlon -O3), but still with --enable-debug
and --enable-cassert.

Sorry, I have no any idea..... Just only full reinstall (with initdb and rm -rf
/usr/local/pgsql) postgresql...
Can you give me login on you computer for a several hours?

--
Teodor Sigaev
teodor@stack.net

Teodor Sigaev <teodor@stack.net> wrote:

Sorry, I have no any idea..... Just only full reinstall (with initdb
and rm -rf /usr/local/pgsql) postgresql...
Can you give me login on you computer for a several hours?

The thing is that when i ran the thing breakpointing on parsetext() at
the line of the call i could type "cont" many times.
Could there be some kind of memory corruption, someone overwriting
memory?

Sorry, i can't hand out a login to the box :(

I did a total re-install just now, and it still crashes.

Magnus