BUG #5773: DEBUG: reaping dead processes DEBUG: server process (PID 10007) was terminated by signal 11: Segme

On 11/27/2010 02:06 AM, Bala Murugan wrote:

The following bug has been logged online:

Bug reference: 5773
Logged by: Bala Murugan
Email address: b2m@a-cti.com
PostgreSQL version: 9.0.1

From ubuntu packages? compiled yourself? from 3rd party repository?

Operating system: Ubuntu

Which ubuntu? 10.04? 10.10?

32-bit x86, 64-bit x64, or other?

Description: DEBUG: reaping dead processes DEBUG: server process
(PID 10007) was terminated by signal 11: Segme
Details:

I am trying to upgrade my postgres version to latest 9.0.1, when I try to
restore server process keep on crash exactly after COPY command.

DEBUG: server process (PID 10007) was terminated by signal 11: Segmentation
fault

It'd be helpful to be able to get a backtrace of this crash. You may
need to install debuginfo packages to do that, or (if you compiled it
yourself) do so with --enable-debug .

http://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD

Alternately, if you're able to post a compressed copy of the data dump
somewhere so someone else can test it that'd be handy. I realize you
probably can't do that, though.

--
Craig Ringer

Thanks for your immediate reply!!. Happy Thanks Giving day!!

postgres@acti-db-bala:~$ uname -a
Linux acti-db-bala 2.6.16.33-xenU #2 SMP Wed Aug 15 17:27:36 SAST 2007
x86_64 GNU/Linux
postgres@acti-db-bala:~$ cat /etc/issue
Ubuntu 9.04 \n \l

Also I used to compile by the below command

./configure --prefix=/var/lib/pgsql --with-perl --with-libxml
--enable-debug

Sorry, I cannot share my backup/dump file, also its about 25Gigs. Please let
me know how to trace this.

Log :

root@acti-db-bala:~# gdb -p 10574
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Attaching to process 10574
Reading symbols from /var/lib/pgsql/bin/postgres...done.
Reading symbols from /usr/lib/libxml2.so.2...Reading symbols from
/usr/lib/debug/usr/lib/libxml2.so.2.6.32...done.
done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...Reading symbols from
/usr/lib/debug/lib/libz.so.1.2.3.3...done.
done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
0x00002ae79777648f in poll () from /lib/libc.so.6
(gdb)
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) set pagination off
(gdb) set logging file /opt/debuglog.txt
(gdb) set logging on
Copying output to /opt/debuglog.txt.
(gdb) cont
Continuing.

Program received signal SIGQUIT, Quit.
0x00002ae79777648f in poll () from /lib/libc.so.6
(gdb)
Continuing.

Program exited normally.
(gdb)

Thanks,
Bala

On Sat, Nov 27, 2010 at 5:26 AM, Craig Ringer
<craig@postnewspapers.com.au>wrote:

On 27/11/2010 5:09 PM, Balamurugan Mahendran wrote:

Program received signal SIGQUIT, Quit.

That's a signal 3 (SIGQUIT). You said you were getting signal 11
(SIGSEGV) crashes. Are you sure this is the same thing?

I suspect you've connected to one of the other backends, not the backend
that actually crashes. You need to identify which one gets the SIGSEGV.
It may be a short-lived autovacuum worker, in which case you'll have a
hard time getting gdb attached to it before it goes splat.

If you do manage to get gdb connected to the crashing backend, instead
of just continuing after gdb pauses at the fatal signal (SIGSEGV), type
at the (gdb) prompt:

bt

then press enter.

You may instead want to enable core dumps as per the instructions on the
same page:

http://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD

... then run gdb on the core dump after the crash. That can potentially
be easier if the crashing backend isn't the one doing the COPY.

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/

On 27/11/2010 5:09 PM, Balamurugan Mahendran wrote:

Thanks for your immediate reply!!. Happy Thanks Giving day!!

postgres@acti-db-bala:~$ uname -a
Linux acti-db-bala 2.6.16.33-xenU #2 SMP Wed Aug 15 17:27:36 SAST 2007
x86_64 GNU/Linux

Is this running on a virtual hosting provider somewhere? If so, which one?

postgres@acti-db-bala:~$ cat /etc/issue
Ubuntu 9.04 \n \l

Also I used to compile by the below command

./configure --prefix=/var/lib/pgsql --with-perl --with-libxml
--enable-debug

... hang on. Did you have anything else in /var/lib/pgsql before running
this install? If you had an old version that you didn't clean out in
there, I'd expect problems.

It's usually wise to install custom-compiled software to a prefix that
doesn't already contain anything else, like --prefix=/opt/pgsql-9.0.1 .
That way it's easy to delete, easy to keep separate from other things, etc.

--
Craig Ringer

Tech-related writing at http://soapyfrogs.blogspot.com/

YES, Its on Amazon EC2 (our production DB running postgres 8.3 version for
more than 3yrs). And its a clean folder.

I am not sure below is the one you are expecting, but hope the below trace
helps..

root@acti-db-bala:~# sudo -u postgres gdb -q -c /var/lib/pgsql/data/core
/var/lib/pgsql/bin/postgres
Reading symbols from /usr/lib/libxml2.so.2...Reading symbols from
/usr/lib/debug/usr/lib/libxml2.so.2.6.32...done.
done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from
/usr/lib/debug/lib/libdl-2.9.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from
/usr/lib/debug/lib/libm-2.9.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...Reading symbols from
/usr/lib/debug/lib/libc-2.9.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...Reading symbols from
/usr/lib/debug/lib/libz.so.1.2.3.3...done.
done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug/lib/ld-2.9.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_files.so.2...Reading symbols from
/usr/lib/debug/lib/libnss_files-2.9.so...done.
done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /var/lib/pgsql/lib/uniqueidentifier.so...done.
Loaded symbols for /var/lib/pgsql/lib/uniqueidentifier.so
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Core was generated by `postgres: postgres acti [local] COPY '.
Program terminated with signal 11, Segmentation fault.
[New process 16223]
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
48 ../sysdeps/x86_64/strlen.S: Permission denied.
in ../sysdeps/x86_64/strlen.S
(gdb)
Current language: auto; currently asm

Thanks,
Bala

On Sat, Nov 27, 2010 at 2:55 PM, Craig Ringer
<craig@postnewspapers.com.au>wrote:

On 11/27/2010 05:45 PM, Balamurugan Mahendran wrote:

YES, Its on Amazon EC2 (our production DB running postgres 8.3 version
for more than 3yrs). And its a clean folder.

OK, thanks.

Core was generated by `postgres: postgres acti [local] COPY '.
Program terminated with signal 11, Segmentation fault.
[New process 16223]
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
48 ../sysdeps/x86_64/strlen.S: Permission denied.
in ../sysdeps/x86_64/strlen.S
(gdb)

Here, you need to type:

bt

and press enter. That requests the backtrace that shows the function
calls leading up to the crash. Without that all we know is that it
crashed in strlen(), which isn't very useful since it was probably a
null pointer dereference caused by an argument passed to it from elsewhere.

--
Craig Ringer

(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
#1 0x0000000000000000 in ?? ()
(gdb)
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
#1 0x0000000000000000 in ?? ()
(gdb)

Thanks,
Bala

On Sat, Nov 27, 2010 at 4:25 PM, Craig Ringer
<craig@postnewspapers.com.au>wrote:

On 11/27/2010 06:58 PM, Balamurugan Mahendran wrote:

(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
#1 0x0000000000000000 in ?? ()
(gdb)
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
#1 0x0000000000000000 in ?? ()

Grr. Corrupt stack. That's pretty much a useless backtrace. Thanks for
trying - sometimes a good backtrace is really useful.

Is there any chance you can simplify your data down to something you can
post? Chop bits out of the schema and COPY data until you find the
smallest SQL script that still crashes?

--
Craig Ringer

I can give you access to my Instance, if you would like to try. you just
need to give me your IP, so that I can white list to allow access.

Thanks,
Bala

On Sat, Nov 27, 2010 at 4:33 PM, Craig Ringer
<craig@postnewspapers.com.au>wrote:

I think I got, what you look for.

root@acti-db-bala:~# sudo -u postgres gdb -q -c /var/lib/pgsql/data/core
/var/lib/pgsql/bin/postgres
Reading symbols from /usr/lib/libxml2.so.2...Reading symbols from
/usr/lib/debug/usr/lib/libxml2.so.2.6.32...done.
done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/libdl.so.2...Reading symbols from
/usr/lib/debug/lib/libdl-2.9.so...done.
done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libm.so.6...Reading symbols from
/usr/lib/debug/lib/libm-2.9.so...done.
done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libc.so.6...Reading symbols from
/usr/lib/debug/lib/libc-2.9.so...done.
done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...Reading symbols from
/usr/lib/debug/lib/libz.so.1.2.3.3...done.
done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/ld-linux-x86-64.so.2...Reading symbols from
/usr/lib/debug/lib/ld-2.9.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_files.so.2...Reading symbols from
/usr/lib/debug/lib/libnss_files-2.9.so...done.
done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /var/lib/pgsql/lib/uniqueidentifier.so...done.
Loaded symbols for /var/lib/pgsql/lib/uniqueidentifier.so
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Core was generated by `postgres: postgres acti [local] COPY '.
Program terminated with signal 11, Segmentation fault.
[New process 32591]
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
48 ../sysdeps/x86_64/strlen.S: Permission denied.
in ../sysdeps/x86_64/strlen.S
(gdb) bt full
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
No locals.
#1 0x00002b08dae084f2 in uniqueidentifier_in () from
/var/lib/pgsql/lib/uniqueidentifier.so
No locals.
#2 0x00000000006ca2ed in InputFunctionCall ()
No locals.
#3 0x000000000050defe in CopyFrom ()
No locals.
#4 0x0000000000510a4e in DoCopy ()
No locals.
#5 0x0000000000612a56 in standard_ProcessUtility ()
No locals.
#6 0x000000000060e907 in PortalRunUtility ()
No locals.
#7 0x000000000060fa7d in PortalRunMulti ()
No locals.
#8 0x0000000000610222 in PortalRun ()
No locals.
#9 0x000000000060c671 in exec_simple_query ()
No locals.
#10 0x000000000060d8c7 in PostgresMain ()
No locals.
#11 0x00000000005dc6ce in ServerLoop ()
No locals.
#12 0x00000000005dd3ff in PostmasterMain ()
No locals.
#13 0x00000000005810e8 in main ()
No locals.
Current language: auto; currently asm
(gdb)

Thanks,
Bala

On Sat, Nov 27, 2010 at 4:36 PM, Balamurugan Mahendran <
balamurugan@adaptavant.com> wrote:

On Sat, Nov 27, 2010 at 12:39, Balamurugan Mahendran <
balamurugan@adaptavant.com> wrote:

I think I got, what you look for.

<snip>

[New process 32591]
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
48 ../sysdeps/x86_64/strlen.S: Permission denied.
in ../sysdeps/x86_64/strlen.S
(gdb) bt full
#0 strlen () at ../sysdeps/x86_64/strlen.S:48
No locals.
#1 0x00002b08dae084f2 in uniqueidentifier_in () from
/var/lib/pgsql/lib/uniqueidentifier.so

Where does your uniqueidentifier.so file come from? It's not a part of
standard PostgreSQL 9.0, and this is the part that's crashing..

Any chance it's a third party module that you didn't recompile for 9.0?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:

Where does your uniqueidentifier.so file come from? It's not a part of
standard PostgreSQL 9.0, and this is the part that's crashing..

There used to be a project of that name on gborg. I can't find the
source code anymore though.

Any chance it's a third party module that you didn't recompile for 9.0?

The magic-block mechanism should prevent that. What I'm wondering about
is whether the input function is (a) careless about null input and (b)
not marked STRICT.

regards, tom lane

On Sat, Nov 27, 2010 at 11:23:46AM -0500, Tom Lane wrote:

Magnus Hagander <magnus@hagander.net> writes:

Where does your uniqueidentifier.so file come from? It's not a part of
standard PostgreSQL 9.0, and this is the part that's crashing..

There used to be a project of that name on gborg. I can't find the
source code anymore though.

How about
http://www.postgresql.org/ftp/projects/gborg/uniqueidentifier/stable/

Any chance it's a third party module that you didn't recompile for 9.0?

The magic-block mechanism should prevent that. What I'm wondering about
is whether the input function is (a) careless about null input and (b)
not marked STRICT.

I think you're right:

PG_FUNCTION_INFO_V1(uniqueidentifier_out);
Datum
uniqueidentifier_out(PG_FUNCTION_ARGS)
{
char *result;
uniqueidentifier *in = (uniqueidentifier *) PG_GETARG_POINTER(0);

if (in == NULL)
PG_RETURN_CSTRING (NULL);

...and...

CREATE OR REPLACE FUNCTION uniqueidentifier_in(cstring)
returns uniqueidentifier
as 'MODULE_PATHNAME'
language 'c';

It should use PG_ARGISNULL(0), no?

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com

Joshua Tolley <eggyknap@gmail.com> writes:

On Sat, Nov 27, 2010 at 11:23:46AM -0500, Tom Lane wrote:

There used to be a project of that name on gborg. I can't find the
source code anymore though.

How about
http://www.postgresql.org/ftp/projects/gborg/uniqueidentifier/stable/

Ah, thanks.

The magic-block mechanism should prevent that. What I'm wondering about
is whether the input function is (a) careless about null input and (b)
not marked STRICT.

I think you're right:

You're looking at the output function not the input function ... and
indeed the first thing the input function does is to invoke strlen(),
without any null check.

It should use PG_ARGISNULL(0), no?

It'd be better to mark it STRICT in the SQL file (and likewise for the
output function). Or actually what he *should* do is drop the whole
thing and use the now-built-in uuid type.

Now, this 2003-vintage tarball is obviously not what the OP is using,
since it hasn't even got a PG_MODULE_MAGIC call. But if it hasn't
been updated any more than that, this'd explain a core dump on NULL
input. What's a bit less clear is how the null got into the source
database without having triggered the same bug; but I suppose it
might've been generated via INSERT DEFAULT VALUES or some such.

regards, tom lane

Thanks all for your help!!

I am not sure that I'll get all my needed function from built-in UUID. I got
the source from the below link also this works perfectly fine with 8.3
version(in my understanding).

http://ring.nict.go.jp/archives/misc/db/postgresql/projects/gborg/uniqueidentifier/stable/uniqueidentifier-0.2.tar.gz

I am very much excited to use UUID, please let me know if there is any link
where i can download and compile all my needed functions, Also I cannot
change my function on my production environment in case needed for uuid
which might needed long hours of testing and hope some of our application
might be in trouble.

Please find the attachment for needed functions.

Again,thanks for everyone who is helping me on this issue.

Thanks,
Bala

On Sun, Nov 28, 2010 at 2:54 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Balamurugan Mahendran <balamurugan@adaptavant.com> writes:

I am not sure that I'll get all my needed function from built-in UUID. I got
the source from the below link also this works perfectly fine with 8.3
version(in my understanding).

http://ring.nict.go.jp/archives/misc/db/postgresql/projects/gborg/uniqueidentifier/stable/uniqueidentifier-0.2.tar.gz

No, it doesn't work "perfectly fine" with 8.3, it's got the same bug it
always had. You were just lucky enough to avoid stumbling over it
before.

I am very much excited to use UUID, please let me know if there is any link
where i can download and compile all my needed functions,

The core uuid type has all that stuff except newid(), which we judged
not to be standardizable. You can find a few different uuid-creation
functions in contrib/uuid-ossp.

regards, tom lane