InitDB: Bad system call

On 9 August 2010 12:56, Torsten Zühlsdorff <foo@meisterderspiele.de> wrote:

Hello,

i've just compiled a new Jail at my FreeBDS 7.0-STABLE machine and trying to
get PostgreSQL 9.0 Beta 4 running. Compiling etc works fine.

But when i call the initdb, i get "Bad System Call" messages. Here is the
output:

$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data -d
Running in debug mode.
VERSION=9.0beta4
PGDATA=/usr/local/pgsql/data
share_path=/usr/local/pgsql/share
PGPATH=/usr/local/pgsql/bin
POSTGRES_SUPERUSERNAME=postgres
POSTGRES_BKI=/usr/local/pgsql/share/postgres.bki
POSTGRES_DESCR=/usr/local/pgsql/share/postgres.description
POSTGRES_SHDESCR=/usr/local/pgsql/share/postgres.shdescription
POSTGRESQL_CONF_SAMPLE=/usr/local/pgsql/share/postgresql.conf.sample
PG_HBA_SAMPLE=/usr/local/pgsql/share/pg_hba.conf.sample
PG_IDENT_SAMPLE=/usr/local/pgsql/share/pg_ident.conf.sample
The files belonging to this database system will be owned by user
"postgres".
This user must also own the server process.

The database cluster will be initialized with locale C.
The default database encoding has accordingly been set to SQL_ASCII.
The default text search configuration will be set to "english".

fixing permissions on existing directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
10
selecting default shared_buffers ... Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
400kB
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... Bad system
call (core dumped)
child process exited with exit code 140
initdb: removing contents of data directory "/usr/local/pgsql/data"

There is no further message in /var/log/messages.

First i believed this is an error relating to SYSVSHM-, SYSVSEM-,
SYSVMSG-options or User-Id (http://www.freebsddiary.org/jail-multiple.php).
But the postgres-user has a user-id which is not used by other
postgres-instances in other jails. And the other options are enabled in the
root-instance.

I also tried to build postgres from a fresh portstree, to make sure, that i
have nothing miss-"./configure"d, but there are the same problems.

I have no clue, what the problem is. Any hints?

Thanks,
Torsten

--

See http://www.postgresql.org/docs/9.0/static/kernel-resources.html
and the section under NetBSD/OpenBSD.

--
Thom Brown
Registered Linux user: #516935

On Mon, Aug 9, 2010 at 6:01 PM, Thom Brown <thom@linux.com> wrote:

See http://www.postgresql.org/docs/9.0/static/kernel-resources.html
and the section under NetBSD/OpenBSD.

--
Thom Brown
Registered Linux user: #516935

Thom

Not sure if it's a typo, but shouldn't he be looking under FreeBSD section
as he is running FreeBSD 7.0?

Amitabh Kant

On 9 August 2010 13:56, Amitabh Kant <amitabhkant@gmail.com> wrote:

On Mon, Aug 9, 2010 at 6:01 PM, Thom Brown <thom@linux.com> wrote:

See http://www.postgresql.org/docs/9.0/static/kernel-resources.html
and the section under NetBSD/OpenBSD.

--
Thom Brown
Registered Linux user: #516935

Thom

Not sure if it's a typo, but shouldn't he be looking under FreeBSD section
as he is running FreeBSD 7.0?

Ah yes, my bad.

--
Thom Brown
Registered Linux user: #516935

Hello Thom,

See http://www.postgresql.org/docs/9.0/static/kernel-resources.html
and the section under NetBSD/OpenBSD.

I already know the FreeBSD section. My current values are:

kern.ipc.shmall: 131072
kern.ipc.shmmax: 2684225436
kern.ipc.semmap: 4096

kern.ipc.semmnu: 512
kern.ipc.semmns: 1024
kern.ipc.semmni: 512

kern.ipc.shm_use_phys: 0

security.jail.sysvipc_allowed: 1

I also run the user with different UIDs:

$ grep pgsql -h /usr/local/jail/*/*/etc/passwd
pgsql:*:1070:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:7575:7575:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1074:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh
pgsql:*:1071:70:PostgreSQL Daemon:/usr/local/pgsql:/bin/sh

I also rebuild the complete jail to make sure, that it is not an error
while creating the jail.
I also disable all - but one (the live-db ;)) - postgresql instance to
make sure, that enough shared memory is free.
But the "bad system call" messages don't go away. Any other hint?

Greetings,
Torsten

Torsten Zï¿œhlsdorff schrieb:

i've just compiled a new Jail at my FreeBDS 7.0-STABLE machine and
trying to get PostgreSQL 9.0 Beta 4 running. Compiling etc works fine.

But when i call the initdb, i get "Bad System Call" messages. Here is
the output:

$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data -d
[output]

First i believed this is an error relating to SYSVSHM-, SYSVSEM-,
SYSVMSG-options or User-Id
(http://www.freebsddiary.org/jail-multiple.php). But the postgres-user
has a user-id which is not used by other postgres-instances in other
jails. And the other options are enabled in the root-instance.

I also tried to build postgres from a fresh portstree, to make sure,
that i have nothing miss-"./configure"d, but there are the same problems.

I've tried the initdb in the only jail PostgreSQL is already running.
There it works.

I have no clue what to do next. I didn't even find the core-dump -.-
Should i just tune-up the System V IPC parameters and hope?

Greetings,
Torsten

--
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse
auswerten kann.

Torsten Zühlsdorff schrieb:

i've just compiled a new Jail at my FreeBDS 7.0-STABLE machine and
trying to get PostgreSQL 9.0 Beta 4 running. Compiling etc works
fine.

Is the machine really running a pre-RELENG 7.0?

But when i call the initdb, i get "Bad System Call" messages. Here
is the output:

The system throwing out a coredump instead of failing gracefully
suggests an OS bug and as you are seemingly running an ancient
development branch, that seems even quite plausible.

In any case I'd ask the same question in the freebsd-questions as
well.

-Reko

Reko Turja schrieb:

i've just compiled a new Jail at my FreeBDS 7.0-STABLE machine and
trying to get PostgreSQL 9.0 Beta 4 running. Compiling etc works fine.

Is the machine really running a pre-RELENG 7.0?

As far as i now, we used the 7.0 versions some month after their
release. So: no.
When i look in, i see in the welcome message:
FreeBSD 7.0-STABLE (GENERIC) #1: Fri Aug 15 19:33:13 CEST 2008

That are 6 months after initial release of 7.0.

But when i call the initdb, i get "Bad System Call" messages. Here is
the output:

The system throwing out a coredump instead of failing gracefully
suggests an OS bug and as you are seemingly running an ancient
development branch, that seems even quite plausible.

I'm running a development *jail* at the *same* machine like the
live-database. The live-database works greats. There is also a second
jail were a postgresql-instance is running. In both i can use Postgresql
(versions 8.3 and 8.4) without any limitations. But in the third-jail i
get the problems.

Greetings,
Torsten

Torsten Zï¿œhlsdorff wrote:

selecting default max_connections ... Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
Bad system call (core dumped)
10
selecting default shared_buffers ... Bad system call (core dumped)
Bad system call (core dumped) ...

What it's doing in this part is trying to start the server process in a
special testing mode, starting with large values for the settings that
impact shared memory, then stepping down the sizes until that works.
That's why there are so many of these. But it looks like none of them
actually work.

Have you tried running the initdb with strace or truss? That might give
you a clue as to exactly what system call is failing. Your jail isn't
allowing something fundamental here, but it's hard to guess what.

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us

Greg Smith <greg@2ndquadrant.com> writes:

Torsten Z�hlsdorff wrote:

Bad system call (core dumped)

Have you tried running the initdb with strace or truss? That might give
you a clue as to exactly what system call is failing. Your jail isn't
allowing something fundamental here, but it's hard to guess what.

Or even easier, gdb the core file ...

regards, tom lane

Hi Tom,

Bad system call (core dumped)

Have you tried running the initdb with strace or truss? That might give
you a clue as to exactly what system call is failing. Your jail isn't
allowing something fundamental here, but it's hard to guess what.

Or even easier, gdb the core file ...

As written early i can't locate the core file. But now i use truss:
$ truss -o /tmp/pg.truss /usr/local/bin/initdb /usr/local/pgsql/

Here is the result:
http://www.dddbl.de/pg.truss.txt

The first suspicious i can see are a lots of "ERR#32 'Broken pipe'" entries.

I also changed some ipc-values from:
kern.ipc.semmni=512
kern.ipc.semmns=1024
kern.ipc.semmnu=512

to:
kern.ipc.semmnu: 4096
kern.ipc.semmns: 8192
kern.ipc.semmni: 32767

But these are read-only values. I have to reboot the machine. But it's a
live-machine and it will take some time to prepare rebooting. -.-

Greetings from Germany,
Torsten

Excerpts from Torsten Zühlsdorff's message of mié ago 11 02:52:34 -0400 2010:

Hi Tom,

Bad system call (core dumped)

Have you tried running the initdb with strace or truss? That might give
you a clue as to exactly what system call is failing. Your jail isn't
allowing something fundamental here, but it's hard to guess what.

Or even easier, gdb the core file ...

As written early i can't locate the core file. But now i use truss:
$ truss -o /tmp/pg.truss /usr/local/bin/initdb /usr/local/pgsql/

This isn't as helpful because you're tracing the initdb process. The
core file would give a backtrace of the postgres process, which is what
is actually crashing.

The first suspicious i can see are a lots of "ERR#32 'Broken pipe'" entries.

This is the result of postgres crashing and thus initdb being unable to
write any more data to it.

I think you should try harder to generate the core file. Maybe you have
too low an "ulimit -c" setting?

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre@commandprompt.com> writes:

Excerpts from Torsten Zühlsdorff's message of mié ago 11 02:52:34 -0400 2010:

Bad system call (core dumped)

I think you should try harder to generate the core file. Maybe you have
too low an "ulimit -c" setting?

The kernel message indicates that core *is* being dumped. Possibly it's
being dumped in the $PGDATA directory, which initdb will rm -rf on
failure. Try using initdb --noclean.

regards, tom lane

Hello,

The first suspicious i can see are a lots of "ERR#32 'Broken pipe'" entries.

This is the result of postgres crashing and thus initdb being unable to
write any more data to it.

I think you should try harder to generate the core file. Maybe you have
too low an "ulimit -c" setting?

There is no ulimit at FreeBSD.

Greetings,
Torsten

Hello,

Excerpts from Torsten ZÌhlsdorff's message of mié ago 11 02:52:34 -0400 2010:

Bad system call (core dumped)

I think you should try harder to generate the core file. Maybe you have
too low an "ulimit -c" setting?

The kernel message indicates that core *is* being dumped. Possibly it's
being dumped in the $PGDATA directory, which initdb will rm -rf on
failure. Try using initdb --noclean.

So... yesterday night i was able to change the SyS-IPC Settings and
restart the server. Good bye 216 days uptime :D

After that i recreate the jail from the scratch and compiled PG 9.0 Beta
4 again. I've compiled PG with:
$ ./configure --enable-debug

InitDB is:
$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data/ --noclean
Running in noclean mode. Mistakes will not be cleaned up.
The files belonging to this database system will be owned by user "pgsql".
This user must also own the server process.

The database cluster will be initialized with locale en_US.ISO8859-1.
The default database encoding has accordingly been set to LATIN1.
The default text search configuration will be set to "english".

creating directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
10
selecting default shared_buffers ... Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
Bad system call
400kB
creating configuration files ... ok
creating template1 database in /usr/local/pgsql/data/base/1 ... Bad
system call
child process exited with exit code 140
initdb: data directory "/usr/local/pgsql/data" not removed at user's request

Result in $PGDATA is:
$ ls -lah /usr/local/pgsql/data/
total 84
drwx------ 12 pgsql pgsql 512B Aug 12 08:56 .
drwx------ 6 pgsql pgsql 512B Aug 12 08:56 ..
-rw------- 1 pgsql pgsql 4B Aug 12 08:56 PG_VERSION
drwx------ 3 pgsql pgsql 512B Aug 12 08:56 base
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 global
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_clog
-rw------- 1 pgsql pgsql 3.8K Aug 12 08:56 pg_hba.conf
-rw------- 1 pgsql pgsql 1.6K Aug 12 08:56 pg_ident.conf
drwx------ 4 pgsql pgsql 512B Aug 12 08:56 pg_multixact
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_notify
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_stat_tmp
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_subtrans
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_tblspc
drwx------ 2 pgsql pgsql 512B Aug 12 08:56 pg_twophase
drwx------ 3 pgsql pgsql 512B Aug 12 08:56 pg_xlog
-rw------- 1 pgsql pgsql 17K Aug 12 08:56 postgresql.conf
-rw------- 1 pgsql pgsql 49B Aug 12 08:56 postmaster.pid

Please notice, that after changing the IPC-Settings of the system, no
core-file is dumped anymore. Quiet interessting.

Greetings,
Torsten

=?ISO-8859-15?Q?Torsten_Z=FChlsdorff?= <foo@meisterderspiele.de> writes:

Please notice, that after changing the IPC-Settings of the system, no
core-file is dumped anymore. Quiet interessting.

How annoying :-(. I think what you need to do is use truss or strace
or local equivalent with the follow-forks flag, so that you can see what
the stand-alone backend process does, not just initdb itself.

regards, tom lane

Hi Tom,

Please notice, that after changing the IPC-Settings of the system, no
core-file is dumped anymore. Quiet interessting.

How annoying :-(. I think what you need to do is use truss or strace
or local equivalent with the follow-forks flag, so that you can see what
the stand-alone backend process does, not just initdb itself.

Ok, next round. I just have truss as an option, because strace didn't
work at my AMD64. Hope its helpfull:

$ truss -f -o /tmp/pgtuss-f.txt /usr/local/pgsql/bin/initdb -D
/usr/local/pgsql/data

Result:
http://www.dddbl.de/pg-truss-f.txt

Greetings,
Torsten

=?ISO-8859-15?Q?Torsten_Z=FChlsdorff?= <foo@meisterderspiele.de> writes:

How annoying :-(. I think what you need to do is use truss or strace
or local equivalent with the follow-forks flag, so that you can see what
the stand-alone backend process does, not just initdb itself.

Ok, next round. I just have truss as an option, because strace didn't
work at my AMD64. Hope its helpfull:

$ truss -f -o /tmp/pgtuss-f.txt /usr/local/pgsql/bin/initdb -D
/usr/local/pgsql/data

Result:
http://www.dddbl.de/pg-truss-f.txt

[ scratches head ... ] That looks like it got interrupted before
getting to anything interesting. Did the console printout show any "Bad
system call" reports?

regards, tom lane

On 12 Aug 2010, at 16:04, Torsten Z�hlsdorff wrote:

Ok, next round. I just have truss as an option, because strace didn't work at my AMD64. Hope its helpfull:

I haven't used it yet, but I've heard good things about DTrace, which is apparently in base these days.

Alban Hertroys

--
Screwing up is an excellent way to attach something to the ceiling.

!DSPAM:737,4c6426a9967631439327345!

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 8/12/10 11:23 AM, Tom Lane wrote:

=?ISO-8859-15?Q?Torsten_Z=FChlsdorff?= <foo@meisterderspiele.de> writes:

How annoying :-(. I think what you need to do is use truss or strace
or local equivalent with the follow-forks flag, so that you can see what
the stand-alone backend process does, not just initdb itself.

Ok, next round. I just have truss as an option, because strace didn't
work at my AMD64. Hope its helpfull:

$ truss -f -o /tmp/pgtuss-f.txt /usr/local/pgsql/bin/initdb -D
/usr/local/pgsql/data

Result:
http://www.dddbl.de/pg-truss-f.txt

[ scratches head ... ] That looks like it got interrupted before
getting to anything interesting. Did the console printout show any "Bad
system call" reports?

Hi,

I didn't see it mentioned earlier in this thread - is
security.jail.sysvipc_allowed=1? This will automatically be set to 1 if
you have jail_sysvipc_allow="YES" in rc.conf.

Regards,

- --
Glen Barber
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQEcBAEBAgAGBQJMZC2yAAoJEFJPDDeguUajRksIAKRDxPxc9MEdo++CVjETSFI6
tRS8uNfnNjLf2DVmY7pAwQfCLvzLRyaJpvpJOeXo76RhqYB79IuRZNODVneXcmUU
6T6KVL+CflR6ql/Vt6XHEdi3VBUCwXmGImxMKm0cN42+cqg9Clr43hPptxTWV0Cw
vv0UIEanS3mTY4yBqwd7gwulLBrFl/X17k1oz8ALRpI+UmMmwEJUkcNANIdbhyrp
7JS0MBVfAO3qXCeG0JeKDvwAmdKOrPUEfumWa8SCqDuLgtK1QT29yEZCf2J2c6vz
jWSalckCQu+Alpse4t42mzC/tyoDBXzPe/zNBd9VRRwQntwnacdjBrjXyR8sv8c=
=6UOg
-----END PGP SIGNATURE-----