Backend died abnormally - postgresql 7.2.1-5

On Tue, Jul 16, 2002 at 09:22:47AM -0500, Rick Eicher II wrote:

NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.

[root@nemisis httpd]# free
total used free shared buffers cached
Mem: 512440 504104 8336 1240 130028
303896
-/+ buffers/cache: 70180 442260
Swap: 80284 312 79972

1. Does this system need more memory?

Doesn't look like it. In general, it might be wise to use a bit more
swap, but that doesn't appear to be causing the problem.

2. What should be my next step in finding this problem?

Is the crash reproducible, and if so, can you post the query or
situation that causes the crash to occur? (you can enable query
logging with debug_print_query in postgresql.conf)

Is there a core file in one of your database directories -- and if
so, can you get a backtrace from it using gdb? It might also be
useful to get a backtrace from a debugging build (--enable-debug).

Cheers,

Neil

--
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

[Cc'ed to -general so that others can help]

On Tue, Jul 16, 2002 at 09:51:48AM -0500, Rick Eicher II wrote:

Is the crash reproducible, and if so, can you post the query or
situation that causes the crash to occur? (you can enable query
logging with debug_print_query in postgresql.conf)

The crash is reproducible. Some examples of a query would be:

Select * from cust_main where last='smith';
Select * from cust_main;

Are there any additional errors in the logs?

With an error that fundamental, I'd suspect hardware problems, namely
bad RAM. Would it be possible to run memtest86 on the machine?

I do have some joins queries but I seem to get this error with any of
query. If I issue the same query four times I will get the error one
time. I have uncommented this line (and others) in postgresql.conf but
do not get any log entries after restart.

Is there a core file in one of your database directories -- and if
so, can you get a backtrace from it using gdb? It might also be
useful to get a backtrace from a debugging build (--enable-debug).

No core file.

Are you sure your system is setup to allow core dumps -- i.e.
does "ulimit -c" produce "unlimited"?

Also, make sure you're looking in the right place for
core files ($PGDATA/base/$oid_of_db/core)

Should I get the source and build it instead of using rpms?

Might be a good bet -- at the least, it should produce a more
helpful backtrace.

Cheers,

Neil

--
Neil Conway <neilconway@rogers.com>
PGP Key ID: DB3C29FC

On Tue, Jul 16, 2002 at 09:51:48AM -0500, Rick Eicher II wrote:

Is the crash reproducible, and if so, can you post the query or
situation that causes the crash to occur? (you can enable query
logging with debug_print_query in postgresql.conf)

The crash is reproducible. Some examples of a query would be:

Select * from cust_main where last='smith';
Select * from cust_main;

Are there any additional errors in the logs?

With an error that fundamental, I'd suspect hardware problems, namely
bad RAM. Would it be possible to run memtest86 on the machine?

I do have some joins queries but I seem to get this error with any

of

query. If I issue the same query four times I will get the error one
time. I have uncommented this line (and others) in postgresql.conf

but

do not get any log entries after restart.

Is there a core file in one of your database directories -- and if
so, can you get a backtrace from it using gdb? It might also be
useful to get a backtrace from a debugging build (--enable-debug).

No core file.

Are you sure your system is setup to allow core dumps -- i.e.
does "ulimit -c" produce "unlimited"?

[root@nemisis root]# ulimit -c
1000000

Also, make sure you're looking in the right place for
core files ($PGDATA/base/$oid_of_db/core)

I was not looking in the right place before. But still no core files
found.

Should I get the source and build it instead of using rpms?

Trying to decide what is the best plan of attack since this is a
production machine. But I think I will run the memtest86 on it first.

There should be more information in the postmaster log than you've shown
us, too. The messages you reported are all from backends *other* than
the one that actually crashed. At the very least the log should have
the postmaster's report of an unexpected child death, with a signal
code.

Also, if you can reproducibly provoke the error, try attaching to the
backend process with gdb before you do so. gdb should be able to give a
backtrace from the crash point even if no core file results.

regards, tom lane