Re: BUG #5862: Postgres dumps core upon a connection attempt

Apologies for lack of detail. Although I've been using Postgres for
years, this is the first time I've had such an issue.

Build options were only --with-perl and --with-python

Below is the output when two different applications attempt to connect
to my 9.0.3 server (note, the second is psql itself):

[root@infinity postgres]# /etc/init.d/archiveopteryx start
Starting Archiveopteryx: aox: Couldn't connect to PostgreSQL. (on
backend 1)
/etc/init.d/archiveopteryx: line 24: 4240 Segmentation
fault /usr/local/archiveopteryx/bin/aox start
done.

[postgres@infinity scripts]$ psql template1
Segmentation fault (core dumped)

Kevin suggested doing a 'make check'. I did so, and it ended with the
following:

mkdir ./testtablespace
./pg_regress --inputdir=. --dlpath=. --multibyte=SQL_ASCII
--temp-install=./tmp_check --top-builddir=../../..
--schedule=./parallel_schedule
make[2]: *** [check] Segmentation fault (core dumped)
make[2]: Leaving directory
`/usr/local/src/postgresql-9.0.3/src/test/regress'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/usr/local/src/postgresql-9.0.3/src/test'
make: *** [check] Error 2

Also, my server doesn't seem to be logging anything, either (although
I'm using the same configuration and start script as 9.0.2)

Lastly, I don't see any 'core' files in the places I would expect.
If/when I find them, I can send along.

- Matt

On 03/02/11 09:53, Matt Zinicola wrote:

Apologies for lack of detail. Although I've been using Postgres for
years, this is the first time I've had such an issue.

Build options were only --with-perl and --with-python

Below is the output when two different applications attempt to connect
to my 9.0.3 server (note, the second is psql itself):

[root@infinity postgres]# /etc/init.d/archiveopteryx start
Starting Archiveopteryx: aox: Couldn't connect to PostgreSQL. (on
backend 1)
/etc/init.d/archiveopteryx: line 24: 4240 Segmentation
fault /usr/local/archiveopteryx/bin/aox start
done.

[postgres@infinity scripts]$ psql template1
Segmentation fault (core dumped)

OK, so it's not the PostgreSQL backend that's crashing, it's psql.

You almost certainly have conflicting libraries lurking around
somewhere, so psql was built against one libpq but lands up getting
linked to another at runtime.

--
System & Network Administrator
POST Newspapers

Hrm.

I did see the Fedora stashed copies of libpq.so.5 and libpq.so.5.2 in
/usr/lib64. I looked everywhere on the system for libpq.so*, and saw that the
only remaining copies where those in my source directory... so I re-built
9.0.3. A 'make check' still died in the same place within the regression
tests. I did a 'make install' anyhow. I cleaned out my data directory and
attempted a new initdb with 9.0.3. That seg faulted as well:

[postgres@infinity local]$ /usr/local/pgsql/bin/initdb -D /data/postgres
Segmentation fault (core dumped)

Any other suggestions?
- Matt

Quoting Craig Ringer <craig@postnewspapers.com.au>:

On 03/02/11 10:33, Matt Zinicola wrote:

Hrm.

I did see the Fedora stashed copies of libpq.so.5 and libpq.so.5.2 in
/usr/lib64. I looked everywhere on the system for libpq.so*, and saw that the
only remaining copies where those in my source directory... so I re-built
9.0.3. A 'make check' still died in the same place within the regression
tests. I did a 'make install' anyhow. I cleaned out my data directory and
attempted a new initdb with 9.0.3. That seg faulted as well:

[postgres@infinity local]$ /usr/local/pgsql/bin/initdb -D /data/postgres
Segmentation fault (core dumped)

What does:

ldd /usr/local/pgsql/bin/initdb

say?

--
System & Network Administrator
POST Newspapers

On 03/02/11 11:11, Matt Zinicola wrote:

OK, it doesn't seem to be a simple problem of linking to the wrong
library then. psql is linking to the correct libpq. initdb isn't linking
to anything much at all, but still crashes for no apparent reason.
Something else may be going on. Please supply the full command line you
used to "./configure" when compiling postgres. If you're not sure what
it was, you can find it in the top of "config.log" in your compile
directory.

Is there any chance you can get us a backtrace of one of the crashing
programs? Try this:

gdb --args psql

Once it loads, it'll drop you to a

(gdb)

prompt. Enter "run" then press enter.

(gdb) run

Psql will then load for a while, crash, and drop you back to a (gdb)
prompt after printing out a message like:

Program received signal SIGSEGV, Segmentation fault.

Enter the "bt" command at the (gdb) prompt and press enter.

(gdb) bt

... then copy and paste everything from "gdb --args psql" through to the
end of the output printed by "bt", put it on http://pastebin.com/ and
send a link to that in your reply email here.

I've created a sample to give you the idea, by starting psql then
intentionally crashing it by sending it a manual SIGSEGV. See:

http://pastebin.com/b8D9i2tb

--
System & Network Administrator
POST Newspapers

As far as the configure options -- Originally, they were merely --with-perl and
--with-python, but just to rule out problems there, I've since just been going
with a straight compile (not additional options).

I will get the backtrace, etc. within the next hour or so. Thanks!

- Matt

Quoting Craig Ringer <craig@postnewspapers.com.au>:

On 02/03/2011 11:15 PM, Matt Zinicola wrote:

I re-compiled with '--enable-debug' and got the symbols. The pastebin is at
http://pastebin.com/xMhEHFdT

That's really interesting. It's getting a NULL path pointer when - I
think - it tries to determine the location of the executables.

Presumably this is something bizarre in your environment - but I have no
idea what it might be. Maybe someone else reading will have an idea.

(Please reply-to-all on further messages so the -bugs list sees things)

--
Craig Ringer

Craig Ringer <craig@postnewspapers.com.au> writes:

On 02/03/2011 11:15 PM, Matt Zinicola wrote:

I re-compiled with '--enable-debug' and got the symbols. The pastebin is at
http://pastebin.com/xMhEHFdT

That's really interesting. It's getting a NULL path pointer when - I
think - it tries to determine the location of the executables.

Hmm ... gdb is evidently lying to us to some extent, because some of
those variables can't possibly be NULL, and control wouldn't have got
to where it says if others of them were. However, it seems clear that
it's dying while trying to determine the actual location of the initdb
executable. Are there any symlinks involved in the path
/usr/local/pgsql/bin/initdb ? Is that located on an unusual filesystem?

regards, tom lane

On Thu, 2011-02-03 at 22:23 -0500, Tom Lane wrote:

Craig Ringer <craig@postnewspapers.com.au> writes:

On 02/03/2011 11:15 PM, Matt Zinicola wrote:

I re-compiled with '--enable-debug' and got the symbols. The pastebin is at
http://pastebin.com/xMhEHFdT

That's really interesting. It's getting a NULL path pointer when - I
think - it tries to determine the location of the executables.

Hmm ... gdb is evidently lying to us to some extent, because some of
those variables can't possibly be NULL, and control wouldn't have got
to where it says if others of them were. However, it seems clear that
it's dying while trying to determine the actual location of the initdb
executable. Are there any symlinks involved in the path
/usr/local/pgsql/bin/initdb ? Is that located on an unusual filesystem?

regards, tom lane

It wasn't an unusual filesystem (other than being within a logical
volume). Nothing out of the ordinary -- a local /ext3 filesystem. I
did a clean re-install of Fedora from scratch, and boom! Postgres
compiled and installed just fine.

Two interesting tidits here (perhaps of note) -- 1) Against my
judgment, I had been using Fedora's upgrade process the last two times I
updated (from F12 to F13, and from F13 to F14). I wonder if that
botched something in my environment and 2) Nothing else on the system
seemed to have trouble (at least up until that point in time). I
suspect it was definitely something underneath Postgres, as when I
deleted everything in /usr/local/pgsql and my cluster (/data/postgres)
and started anew, it still had the very same problem. Just some
additional points of info.

In any case... a "clean" install of Fedora 14 did not yield this
problem, so feel free to close this issue if/when you feel appropriate.

Thanks to everyone that was lending assistance. It's much appreciated.

- Matt

On 04/02/11 15:11, Craig Ringer wrote:

On 02/03/2011 11:15 PM, Matt Zinicola wrote:

I re-compiled with '--enable-debug' and got the symbols. The
pastebin is at
http://pastebin.com/xMhEHFdT

That's really interesting. It's getting a NULL path pointer when - I
think - it tries to determine the location of the executables.

Presumably this is something bizarre in your environment - but I have
no idea what it might be. Maybe someone else reading will have an idea.

(Coming in too late, but...)

I'd be interested to see what happens if you do:

$ export PATH=/usr/local/pgsql/bin:$PATH
$ export LD_LIBRARY_PATH=/usr/local/pgsql/lib
$ initdb -D /data/postgres
$ pg_ctl -D /data/postgres start;
$ psql

I'm guessing that there are older libraries or binaries earlier in your
various env paths, and these are tripping up postgres.

Cheers

Mark

Matthew Zinicola <matt@zinicola.com> writes:

It wasn't an unusual filesystem (other than being within a logical
volume). Nothing out of the ordinary -- a local /ext3 filesystem. I
did a clean re-install of Fedora from scratch, and boom! Postgres
compiled and installed just fine.

Two interesting tidits here (perhaps of note) -- 1) Against my
judgment, I had been using Fedora's upgrade process the last two times I
updated (from F12 to F13, and from F13 to F14). I wonder if that
botched something in my environment and 2) Nothing else on the system
seemed to have trouble (at least up until that point in time).

Hmm. Given that you couldn't reproduce it on a clean system, I'd have
to agree that it sounds like something was a bit wacko about the
upgraded system. One does hear of people having trouble with that
process from time to time.

regards, tom lane