Unhelpful initdb error message

Thom Brown <thom@linux.com> writes:

thom@swift:~/Development$ initdb
The files belonging to this database system will be owned by user "thom".
This user must also own the server process.

The database cluster will be initialized with locale en_GB.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /home/thom/Development/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
creating configuration files ... ok
creating template1 database in /home/thom/Development/data/base/1 ...
FATAL: could not remove old lock file "postmaster.pid": No such file
or directory
HINT: The file seems accidentally left over, but it could not be
removed. Please remove the file by hand and try again.
child process exited with exit code 1
initdb: removing contents of data directory "/home/thom/Development/data"

Um ... I assume this is some patched version rather than pristine
sources? It's pretty hard to explain why it's falling over like that.

I don't think there is anything wrong with the error message, because
it's intended for the case where some previous postmaster failed and
left a lock file behind. The question is how is it you're getting to
that error, not whether we should change its text.

One possible lead is that it looks like the postmaster-starting probes
to select max_connections and shared_buffers all failed too, since those
numbers came out as the minimums.

regards, tom lane

On Tuesday, March 06, 2012 7:46:37 am Thom Brown wrote:

Hi all,

After building Postgres and trying an initdb, I'm getting the following:

thom@swift:~/Development$ initdb
The files belonging to this database system will be owned by user "thom".
This user must also own the server process.

The database cluster will be initialized with locale en_GB.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /home/thom/Development/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
creating configuration files ... ok
creating template1 database in /home/thom/Development/data/base/1 ...
FATAL: could not remove old lock file "postmaster.pid": No such file
or directory
HINT: The file seems accidentally left over, but it could not be
removed. Please remove the file by hand and try again.
child process exited with exit code 1
initdb: removing contents of data directory "/home/thom/Development/data"

It can't remove an old lock file due to it not existing, but the hint
says it was left over but couldn't be removed. The hint contradicts
the error message. There is nothing in the data directory at all
before trying this, and nothing after. Repeating initdb yields the
same result.

But, if I rename the data directory to something else and mkdir data
again, all is well. I can make it break again by removing the new
data directory and renaming the old one back to data, still completely
empty. Note that throughout all of this, Postgres is running, but as
a separate user and using completely separate directories, since it's
the standard packaged version on Debian.

Can anyone suggest what is wrong here?

The postmaster.pid is located outside the data directory, but points back to the
data directory. Not sure where Debian, though at a guess somewhere in /var.
Any way search for postmaster.pid.

--
Adrian Klaver
adrian.klaver@gmail.com

On 6 March 2012 16:02, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thom Brown <thom@linux.com> writes:

thom@swift:~/Development$ initdb
The files belonging to this database system will be owned by user "thom".
This user must also own the server process.

The database cluster will be initialized with locale en_GB.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

fixing permissions on existing directory /home/thom/Development/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 10
selecting default shared_buffers ... 400kB
creating configuration files ... ok
creating template1 database in /home/thom/Development/data/base/1 ...
FATAL:  could not remove old lock file "postmaster.pid": No such file
or directory
HINT:  The file seems accidentally left over, but it could not be
removed. Please remove the file by hand and try again.
child process exited with exit code 1
initdb: removing contents of data directory "/home/thom/Development/data"

Um ... I assume this is some patched version rather than pristine
sources?  It's pretty hard to explain why it's falling over like that.

No, I did a "git stash", "git clean -f" and "git pull" before trying to build.

--
Thom

On 6 March 2012 16:04, Adrian Klaver <adrian.klaver@gmail.com> wrote:

The postmaster.pid is located outside the data directory, but points back to the
data directory.   Not sure where Debian, though at a guess somewhere in /var.
Any way search for postmaster.pid.

I'm not sure, because if I use a new data directory, initdb it and
start the service, the postmaster.pid appears in it, and not as a
symbolic link.

I did a search for postmaster.pid in the whole of /var and it only
shows up "/var/lib/postgresql/9.1/main/postmaster.pid"

--
Thom

On 6 March 2012 16:11, Thom Brown <thom@linux.com> wrote:

On 6 March 2012 16:04, Adrian Klaver <adrian.klaver@gmail.com> wrote:

The postmaster.pid is located outside the data directory, but points back to the
data directory.   Not sure where Debian, though at a guess somewhere in /var.
Any way search for postmaster.pid.

I'm not sure, because if I use a new data directory, initdb it and
start the service, the postmaster.pid appears in it, and not as a
symbolic link.

I did a search for postmaster.pid in the whole of /var and it only
shows up "/var/lib/postgresql/9.1/main/postmaster.pid"

Correction, this is Ubuntu, not Debian. 11.10 if it's of any consequence.

The file system is ext4 with
rw,noatime,nodiratime,errors=remount-ro,commit=0 on a Crucial m4 SSD.

ecryptfs is in use in the parent directory.

--
Thom

On Tuesday, March 06, 2012 8:11:20 am Thom Brown wrote:

On 6 March 2012 16:04, Adrian Klaver <adrian.klaver@gmail.com> wrote:

The postmaster.pid is located outside the data directory, but points back
to the data directory. Not sure where Debian, though at a guess
somewhere in /var. Any way search for postmaster.pid.

I'm not sure, because if I use a new data directory, initdb it and
start the service, the postmaster.pid appears in it, and not as a
symbolic link.

I did a search for postmaster.pid in the whole of /var and it only
shows up "/var/lib/postgresql/9.1/main/postmaster.pid"

My guess is if you open that file you will find it points back to the old
directory. So are you still running the Debian packaged version of Postgres?
Or in other words does a ps show any other postmasters running other than the
new one you built?

--
Adrian Klaver
adrian.klaver@gmail.com

On 6 March 2012 16:18, Adrian Klaver <adrian.klaver@gmail.com> wrote:

On Tuesday, March 06, 2012 8:11:20 am Thom Brown wrote:

On 6 March 2012 16:04, Adrian Klaver <adrian.klaver@gmail.com> wrote:

The postmaster.pid is located outside the data directory, but points back
to the data directory.   Not sure where Debian, though at a guess
somewhere in /var. Any way search for postmaster.pid.

I'm not sure, because if I use a new data directory, initdb it and
start the service, the postmaster.pid appears in it, and not as a
symbolic link.

I did a search for postmaster.pid in the whole of /var and it only
shows up "/var/lib/postgresql/9.1/main/postmaster.pid"

My guess is if you open that file you will find it points back to the old
directory.  So are you  still running the Debian packaged version of Postgres?
Or in other words does a ps show any other postmasters running other than the
new one you built?

No, only the ones running as the postgres user.

Here's the contents of the pid file in /var/lib/postgresql/9.1/main/

1199
/var/lib/postgresql/9.1/main
1330883367
5432
/var/run/postgresql
localhost
5432001 0

And if I start my development copy, this is the content of its postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
5488001 191365126

--
Thom

Thom Brown <thom@linux.com> writes:

On 6 March 2012 16:02, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Um ... I assume this is some patched version rather than pristine
sources? �It's pretty hard to explain why it's falling over like that.

No, I did a "git stash", "git clean -f" and "git pull" before trying to build.

[ scratches head... ] I can't reproduce it with current git tip.

regards, tom lane

On 6 March 2012 16:31, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thom Brown <thom@linux.com> writes:

On 6 March 2012 16:02, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Um ... I assume this is some patched version rather than pristine
sources?  It's pretty hard to explain why it's falling over like that.

No, I did a "git stash", "git clean -f" and "git pull" before trying to build.

[ scratches head... ]  I can't reproduce it with current git tip.

And I don't think I can reproduce this if I remove that directory.
I've seen this issue about 3 or 4 times in the past, and fixed it by
ditching the old data dir completely. I'm just not sure what causes
this to happen.

Looking back through my terminal log, one thing might lend a clue from
before I tried rebuliding it:

thom@swift:~/Development$ pg_ctl stop
waiting for server to shut down....cd .postgre.s
.............
....

....^C
thom@swift:~/Development$ pg_ctl stop
pg_ctl: could not send stop signal (PID: 2807): No such process
thom@swift:~/Development$ ps -ef | grep postgres
postgres 1199 1 0 Mar04 ? 00:00:01
/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main
-c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres 1273 1199 0 Mar04 ? 00:00:18 postgres: writer
process
postgres 1274 1199 0 Mar04 ? 00:00:14 postgres: wal writer
process
postgres 1275 1199 0 Mar04 ? 00:00:03 postgres: autovacuum
launcher process
postgres 1276 1199 0 Mar04 ? 00:00:02 postgres: stats
collector process
thom 16476 4302 0 15:30 pts/1 00:00:00 grep --color=auto postgres

Postgres wouldn't shut down. I had no other terminal windows using
psql, no other database client apps open, yet it stayed shutting down,
so I CTRL+C'd it and tried again. A quick check of running processes
showed that it had stopped running. (it shows postgres running above,
but the dev copy runs as my user, not postgres)

--
Thom

On Tuesday, March 06, 2012 8:24:20 am Thom Brown wrote:

No, only the ones running as the postgres user.

In my original read, I missed the part you had the Ubuntu/Debian packaged
version running.

Here's the contents of the pid file in /var/lib/postgresql/9.1/main/

1199
/var/lib/postgresql/9.1/main
1330883367
5432
/var/run/postgresql
localhost
5432001 0

And if I start my development copy, this is the content of its
postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
5488001 191365126

So how are getting the file above? I thought initdb refused to init the directory
and that you could not find pid file it was referring to? Just on a hunch, what is
in /tmp?

--
Adrian Klaver
adrian.klaver@gmail.com

On 6 March 2012 16:40, Adrian Klaver <adrian.klaver@gmail.com> wrote:

On Tuesday, March 06, 2012 8:24:20 am Thom Brown wrote:

No, only the ones running as the postgres user.

In my original read, I missed the part you had the Ubuntu/Debian packaged
version running.

Here's the contents of the pid file in /var/lib/postgresql/9.1/main/

1199
/var/lib/postgresql/9.1/main
1330883367
5432
/var/run/postgresql
localhost
  5432001         0

And if I start my development copy, this is the content of its
postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
  5488001 191365126

So how are getting the file above? I thought initdb refused to init the directory
and that you could not find pid file it was referring to? Just on a hunch, what is
in /tmp?

I got the above output when I created a new data directory and initdb'd it.

/tmp shows:

4 -rw------- 1 thom thom 55 2012-03-06 16:22
.s.PGSQL.5488.lock
0 srwxrwxrwx 1 thom thom 0 2012-03-06 16:22 .s.PGSQL.5488

Once it's up and running. These disappear after though. When using
the old data directory again, there's no evidence of anything like
this in /tmp.

--
Thom

On Tuesday, March 06, 2012 8:44:10 am Thom Brown wrote:

And if I start my development copy, this is the content of its
postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
5488001 191365126

So how are getting the file above? I thought initdb refused to init the
directory and that you could not find pid file it was referring to? Just
on a hunch, what is in /tmp?

I got the above output when I created a new data directory and initdb'd it.

Still not understanding. In your original post you said
/home/thom/Development/data was the original directory you could not initdb. How
could it also be the new directory you can initdb as indicated by the
postmaster.pid?

From your previous post:
thom@swift:~/Development$ pg_ctl stop
pg_ctl: could not send stop signal (PID: 2807): No such process

Doing the above without qualifying which version of pg_ctl you are using or what
data directory you are pointing is dangerous. The combination of implied
pathing and preset env variables could lead to all sorts of mischief.

/tmp shows:

4 -rw------- 1 thom thom 55 2012-03-06 16:22
.s.PGSQL.5488.lock
0 srwxrwxrwx 1 thom thom 0 2012-03-06 16:22
.s.PGSQL.5488

Once it's up and running. These disappear after though. When using
the old data directory again, there's no evidence of anything like
this in /tmp.

--
Adrian Klaver
adrian.klaver@gmail.com

On 6 March 2012 17:00, Adrian Klaver <adrian.klaver@gmail.com> wrote:

On Tuesday, March 06, 2012 8:44:10 am Thom Brown wrote:

And if I start my development copy, this is the content of its
postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
  5488001 191365126

So how are getting the file above? I thought initdb refused to init the
directory and that you could not find pid file it was referring to? Just
on a hunch, what is in /tmp?

I got the above output when I created a new data directory and initdb'd it.

Still not understanding. In your original post you said
/home/thom/Development/data was the original directory you could not initdb. How
could it also be the new directory you can initdb as indicated by the
postmaster.pid?

/home/thom/Development/data was causing problems so:

mv data databroken
mkdir data
initdb

... working fine again. I then used the postmaster.pid from this when
started up. But if I do:

pg_ctl stop
rm -rf data
mv databroken data
initdb

... error messages appear again.

From your previous post:
 thom@swift:~/Development$ pg_ctl stop
 pg_ctl: could not send stop signal (PID: 2807): No such process

Doing the above without qualifying which version of pg_ctl you are using or what
data directory you are pointing is dangerous.  The combination of  implied
pathing and preset env variables could lead to all sorts of mischief.

Unlikely since pg_ctl isn't available in my search path once I remove
my local development bin dir from it. All non-client tools for the
packaged version aren't available to normal users. Those are all in
/usr/lib/postgresql/9.1/bin. The only ones exposed to the search path
through symbolic links are:

clusterdb
createdb
createlang
createuser
dropdb
droplang
dropuser
pg_dump
pg_dumpall
pg_restore
psql
reindexdb
vacuumdb
vacuumlo

--
Thom

Thom Brown <thom@linux.com> writes:

Looking back through my terminal log, one thing might lend a clue from
before I tried rebuliding it:

thom@swift:~/Development$ pg_ctl stop
waiting for server to shut down....cd .postgre.s
.............
....

....^C
thom@swift:~/Development$ pg_ctl stop
pg_ctl: could not send stop signal (PID: 2807): No such process
thom@swift:~/Development$ ps -ef | grep postgres
postgres 1199 1 0 Mar04 ? 00:00:01
/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main
-c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres 1273 1199 0 Mar04 ? 00:00:18 postgres: writer
process
postgres 1274 1199 0 Mar04 ? 00:00:14 postgres: wal writer
process
postgres 1275 1199 0 Mar04 ? 00:00:03 postgres: autovacuum
launcher process
postgres 1276 1199 0 Mar04 ? 00:00:02 postgres: stats
collector process
thom 16476 4302 0 15:30 pts/1 00:00:00 grep --color=auto postgres

Hm. It looks like pg_ctl found a PID file pointing to a non-existent
process, which is a bit like what you're seeing initdb do.

I wonder whether this is somehow caused by conflicting settings for
PGDATA. Do you have a setting for that in your environment, or .bashrc
or someplace, that is different from what you're trying to use?

regards, tom lane

On Tuesday, March 06, 2012 9:09:41 am Thom Brown wrote:

On 6 March 2012 17:00, Adrian Klaver <adrian.klaver@gmail.com> wrote:

On Tuesday, March 06, 2012 8:44:10 am Thom Brown wrote:

And if I start my development copy, this is the content of its
postmaster.pid:

27061
/home/thom/Development/data
1331050950
5488
/tmp
localhost
5488001 191365126

So how are getting the file above? I thought initdb refused to init
the directory and that you could not find pid file it was referring
to? Just on a hunch, what is in /tmp?

I got the above output when I created a new data directory and initdb'd
it.

Still not understanding. In your original post you said
/home/thom/Development/data was the original directory you could not
initdb. How could it also be the new directory you can initdb as
indicated by the postmaster.pid?

/home/thom/Development/data was causing problems so:

mv data databroken
mkdir data
initdb

... working fine again. I then used the postmaster.pid from this when
started up. But if I do:

pg_ctl stop
rm -rf data
mv databroken data
initdb

... error messages appear again.

Humph, need more coffee.

From your previous post:
thom@swift:~/Development$ pg_ctl stop
pg_ctl: could not send stop signal (PID: 2807): No such process

Doing the above without qualifying which version of pg_ctl you are using
or what data directory you are pointing is dangerous. The combination
of implied pathing and preset env variables could lead to all sorts of
mischief.

Unlikely since pg_ctl isn't available in my search path once I remove
my local development bin dir from it. All non-client tools for the
packaged version aren't available to normal users. Those are all in
/usr/lib/postgresql/9.1/bin. The only ones exposed to the search path
through symbolic links are:

env variables?

--
Adrian Klaver
adrian.klaver@gmail.com

On 6 March 2012 17:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thom Brown <thom@linux.com> writes:

Looking back through my terminal log, one thing might lend a clue from
before I tried rebuliding it:

thom@swift:~/Development$ pg_ctl stop
waiting for server to shut down....cd .postgre.s
.............
....

....^C
thom@swift:~/Development$ pg_ctl stop
pg_ctl: could not send stop signal (PID: 2807): No such process
thom@swift:~/Development$ ps -ef | grep postgres
postgres  1199     1  0 Mar04 ?        00:00:01
/usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main
-c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres  1273  1199  0 Mar04 ?        00:00:18 postgres: writer
process
postgres  1274  1199  0 Mar04 ?        00:00:14 postgres: wal writer
process
postgres  1275  1199  0 Mar04 ?        00:00:03 postgres: autovacuum
launcher process
postgres  1276  1199  0 Mar04 ?        00:00:02 postgres: stats
collector process
thom     16476  4302  0 15:30 pts/1    00:00:00 grep --color=auto postgres

Hm.  It looks like pg_ctl found a PID file pointing to a non-existent
process, which is a bit like what you're seeing initdb do.

I wonder whether this is somehow caused by conflicting settings for
PGDATA.  Do you have a setting for that in your environment, or .bashrc
or someplace, that is different from what you're trying to use?

These are in my env output:

PATH=/home/thom/Development/psql/bin/:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PGDATA=/home/thom/Development/data/
PGPORT=5488

This appears in my build script before configure:

export PGDATA=$HOME/Development/data/
export PATH=$HOME/Development/psql/bin/:$PATH
export PGPORT=5488

And those 3 lines also appear in my .bashrc file without any variation:

export PGDATA=$HOME/Development/data/
export PATH=$HOME/Development/psql/bin/:$PATH
export PGPORT=5488

--
Thom

Adrian Klaver <adrian.klaver@gmail.com> writes:

The postmaster.pid is located outside the data directory, but points back to the
data directory. Not sure where Debian, though at a guess somewhere in /var.
Any way search for postmaster.pid.

Really? That seems like an extremely dangerous/stupid/unnecessary hack
on the part of the Debian packagers. What's keeping users from
accidentally starting two postmasters in the same data directory, if
they can put their pidfiles in (different) other places?

(This seems unrelated to Thom's issue, but it's still worrisome.)

regards, tom lane

On Tuesday, March 06, 2012 9:25:17 am Thom Brown wrote:

These are in my env output:

PATH=/home/thom/Development/psql/bin/:/usr/lib/lightdm/lightdm:/usr/local/s
bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
PGDATA=/home/thom/Development/data/
PGPORT=5488

This appears in my build script before configure:

export PGDATA=$HOME/Development/data/
export PATH=$HOME/Development/psql/bin/:$PATH
export PGPORT=5488

And those 3 lines also appear in my .bashrc file without any variation:

export PGDATA=$HOME/Development/data/
export PATH=$HOME/Development/psql/bin/:$PATH
export PGPORT=5488

And you are sure there is no pg_ctl or initdb outside
/usr/lib/postgresql/9.1/bin or /home/thom/Development/psql/bin and in your PATH?

Just for grins what happens if you try an initdb using an explicit reference to
the binary /home/thom/Development/psql/bin/initdb and the -D
/home/thom/Development/data/ ?

--
Adrian Klaver
adrian.klaver@gmail.com

Thom Brown <thom@linux.com> writes:

On 6 March 2012 16:31, Tom Lane <tgl@sss.pgh.pa.us> wrote:

[ scratches head... ] �I can't reproduce it with current git tip.

And I don't think I can reproduce this if I remove that directory.
I've seen this issue about 3 or 4 times in the past, and fixed it by
ditching the old data dir completely. I'm just not sure what causes
this to happen.

I'm a bit confused here. Isn't the data directory totally empty before
initdb starts? It's supposed to refuse to proceed otherwise.

regards, tom lane