postgresql doesn't start

Paolo De Michele <paolo@paolodemichele.it> writes:

2015-10-01 21:40:20 UTC FATAL: could not remove old lock file
"postmaster.pid": Permission denied

Looks like something removed the postmaster's write permission on
the data directory itself.

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/2/2015 2:02 PM, Paolo De Michele wrote:

I've postgresql 9.3 installed on docker

docker is a packaging system for linux containers, its not an operating
system per se. what OS is your docker container running on?
supervisord is a process/daemon manager, its not part of 'postgresql',
although apparently your docker container is using it to run postgresql.

your questions all seem related to the specific packaging of this
particular docker container, you might see who created this container
configuration, and ask them these questions as there's a lot going on
there abstracting things.

--
john r pierce, recycling bits in santa cruz

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/02/2015 02:02 PM, Paolo De Michele wrote:

hi there,

I've a big problem with my postgresql installation
I've postgresql 9.3 installed on docker; I start it via supervisord
I've never had issues with postgresql and I don't touched nothing (no
update, no changes)

this is my configuration
there's a file called postgresql.conf in /etc/supervisor/conf.d/
cat command:

[program:postgres]
command=/opt/postgresql.sh
autostart=true
autorestart=true
stopsignal=QUIT

If it where me I would change the above to:

[program:postgres]
command=/opt/postgresql.sh
autostart=true
stopsignal=TERM

lookt at /opt/postgresql.sh

#!/bin/sh

# This script is run by Supervisor to start PostgreSQL 9.3 in foreground
mode

if [ -d /var/run/postgresql ]; then
chmod 2775 /var/run/postgresql
else
install -d -m 2775 -o postgres -g postgres /var/run/postgresql
fi

exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main -c
config_file=/etc/postgresql/9.3/main/postgresql.conf"

until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:

2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over,
but it could not be removed. Please remove the file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock file
"postmaster.pid": Permission denied

if I remove this file with sudo permissions when I re-run the process
I've the same error and I don't understand why. I tried the same
configuration (postgresql installation and configuration) in another new
docker installation and works fine.

what's the problem?
someone help me?
I've very important databases and I dont' know how to recover it

please let me know, thanks in advance

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/2/2015 2:02 PM, Paolo De Michele wrote:

exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main -c
config_file=/etc/postgresql/9.3/main/postgresql.conf"

until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:

2015-10-01 21:40:18 UTC HINT: The file seems accidentally left over,
but it could not be removed. Please remove the file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock file
"postmaster.pid": Permission denied

try...
ls -la /var/lib/postgresql/9.3/main

The directory . should be owned by the postgres user, and it should have
700, 750, or 770 permissions. all the files in it should also be owned
by postgres.

also look and see if postgres logged anything in its own system log
files (/var/log/postgresql/9.3 or whatever).

--
john r pierce, recycling bits in santa cruz

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

​Are you using docker on centos ? I had problem with
centos/docker/postgresql because container size was (maybe still is)
limited to 20GB on that specific OS​. Maybe not related, but good to know

2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com>:

hi there,

thanks a lot
so, I'm using docker on ubuntu 14.04
about permissions:

# ls -la /var/lib/postgresql/9.3/main
total 72
drwx------ 28 postgres postgres 4096 Oct 3 12:41 .
drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 ..
-rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION
drwx------ 12 postgres postgres 4096 Oct 1 11:59 base
drwx------ 2 postgres postgres 4096 Oct 1 11:59 global
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog
drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase
drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog
-rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts
-rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid

in /var/log/postgresql/ I've a file with 0kb
it is weird
is there a way to recover all dbs without starting the daemon?
because I've another one docker with the same configuration and works fine
(same os, same versione of postgresql)

On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com> wrote:

On 10/03/2015 05:50 AM, Paolo De Michele wrote:

hi there,

thanks a lot
so, I'm using docker on ubuntu 14.04
about permissions:

# ls -la /var/lib/postgresql/9.3/main
total 72
drwx------ 28 postgres postgres 4096 Oct 3 12:41 .
drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 ..
-rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION
drwx------ 12 postgres postgres 4096 Oct 1 11:59 base
drwx------ 2 postgres postgres 4096 Oct 1 11:59 global
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog
drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase
drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog
-rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts
-rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid

in /var/log/postgresql/ I've a file with 0kb
it is weird
is there a way to recover all dbs without starting the daemon?

What OS and version are you running? Assuming a Debian/Ubuntu flavor for
now.

How was Postgres installed?

So there is a pid file present in the data directory, how about in the
run directory, /var/run/postgresql?

Is Postgres running? so:

ps ax|grep post

or

pg_ctl status -D /var/lib/postgresql/9.3/main

If not what happens if you remove the pid and start Postgres manually,
not through supervisor?

because I've another one docker with the same configuration and works
fine (same os, same versione of postgresql)

On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com
<mailto:niparisco@gmail.com>> wrote:

​Are you using docker on centos ? I had problem with
centos/docker/postgresql because container size was (maybe still is)
limited to 20GB on that specific OS​. Maybe not related, but good to
know

2015-10-03 0:03 GMT+02:00 John R Pierce <pierce@hogranch.com
<mailto:pierce@hogranch.com>>:

On 10/2/2015 2:02 PM, Paolo De Michele wrote:

exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main -c
config_file=/etc/postgresql/9.3/main/postgresql.conf"

until yesterday there were no problem
right now I see this in the /var/log/supervisor's directory:

2015-10-01 21:40:18 UTC HINT: The file seems accidentally
left over, but it could not be removed. Please remove the
file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove old lock
file "postmaster.pid": Permission denied

try...
ls -la /var/lib/postgresql/9.3/main

The directory . should be owned by the postgres user, and it
should have 700, 750, or 770 permissions. all the files in it
should also be owned by postgres.

also look and see if postgres logged anything in its own system
log files (/var/log/postgresql/9.3 or whatever).

--
john r pierce, recycling bits in santa cruz

--
Sent via pgsql-general mailing list
(pgsql-general@postgresql.org <mailto:pgsql-general@postgresql.org>)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Adrian,

thanks a lot
about your questions

1. postgresql was installed via apt-get install postgresql -y command
2. in /var/run/postgresql/ there is only one file called 9.3-main.pid
3. postgres is not running, I did not find nothing with ps command
4. I remove the postmaster.pid file and I tried to run postgresql manually.
this is the result:

# /etc/init.d/postgresql start
* Starting PostgreSQL 9.3 database server
The PostgreSQL server failed to start. Please check the log output.
...fail!

I verified in /var/log/postgresql/ and I've not found nothing

On 3 October 2015 at 15:19, Adrian Klaver <adrian.klaver@aklaver.com> wrote:

On 10/04/2015 02:45 AM, Paolo De Michele wrote:

Adrian,

thanks a lot
about your questions

1. postgresql was installed via apt-get install postgresql -y command

What OS and version?

2. in /var/run/postgresql/ there is only one file called 9.3-main.pid
3. postgres is not running, I did not find nothing with ps command
4. I remove the postmaster.pid file and I tried to run postgresql

Just the postmaster.pid or it and the 9.3-main.pid?

manually. this is the result:

# /etc/init.d/postgresql start
* Starting PostgreSQL 9.3 database server
The PostgreSQL server failed to start. Please check the log output.
...fail!

Try:

pg_ctlcluster 9.3 main start

I verified in /var/log/postgresql/ and I've not found nothing

Nothing as no text at all or nothing as no messages that apply?

What does the system log show?

On 3 October 2015 at 15:19, Adrian Klaver <adrian.klaver@aklaver.com
<mailto:adrian.klaver@aklaver.com>> wrote:

On 10/03/2015 05:50 AM, Paolo De Michele wrote:

hi there,

thanks a lot
so, I'm using docker on ubuntu 14.04
about permissions:

# ls -la /var/lib/postgresql/9.3/main
total 72
drwx------ 28 postgres postgres 4096 Oct 3 12:41 .
drwxr-xr-x 4 postgres postgres 4096 Oct 1 11:59 ..
-rwx------ 1 postgres postgres 4 Mar 21 2015 PG_VERSION
drwx------ 12 postgres postgres 4096 Oct 1 11:59 base
drwx------ 2 postgres postgres 4096 Oct 1 11:59 global
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_clog
drwx------ 6 postgres postgres 4096 Oct 1 11:59 pg_multixact
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_notify
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_serial
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_snapshots
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_stat
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Oct 1 11:59 pg_subtrans
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_tblspc
drwx------ 2 postgres postgres 4096 Mar 21 2015 pg_twophase
drwx------ 4 postgres postgres 4096 Oct 1 12:18 pg_xlog
-rwx------ 1 postgres postgres 133 Oct 1 20:00 postmaster.opts
-rw------- 1 postgres postgres 50 Oct 3 12:41 postmaster.pid

in /var/log/postgresql/ I've a file with 0kb
it is weird
is there a way to recover all dbs without starting the daemon?

What OS and version are you running? Assuming a Debian/Ubuntu flavor
for now.

How was Postgres installed?

So there is a pid file present in the data directory, how about in
the run directory, /var/run/postgresql?

Is Postgres running? so:

ps ax|grep post

or

pg_ctl status -D /var/lib/postgresql/9.3/main

If not what happens if you remove the pid and start Postgres
manually, not through supervisor?

because I've another one docker with the same configuration and
works
fine (same os, same versione of postgresql)

On 3 October 2015 at 07:13, Nicolas Paris <niparisco@gmail.com
<mailto:niparisco@gmail.com>
<mailto:niparisco@gmail.com <mailto:niparisco@gmail.com>>> wrote:

​Are you using docker on centos ? I had problem with
centos/docker/postgresql because container size was (maybe
still is)
limited to 20GB on that specific OS​. Maybe not related,
but good to
know

2015-10-03 0:03 GMT+02:00 John R Pierce
<pierce@hogranch.com <mailto:pierce@hogranch.com>
<mailto:pierce@hogranch.com <mailto:pierce@hogranch.com>>>:

On 10/2/2015 2:02 PM, Paolo De Michele wrote:

exec su postgres -c
"/usr/lib/postgresql/9.3/bin/postgres -D
/var/lib/postgresql/9.3/main -c
config_file=/etc/postgresql/9.3/main/postgresql.conf"

until yesterday there were no problem
right now I see this in the /var/log/supervisor's
directory:

2015-10-01 21:40:18 UTC HINT: The file seems
accidentally
left over, but it could not be removed. Please
remove the
file by hand and try again.
2015-10-01 21:40:20 UTC FATAL: could not remove
old lock
file "postmaster.pid": Permission denied

try...
ls -la /var/lib/postgresql/9.3/main

The directory . should be owned by the postgres user,
and it
should have 700, 750, or 770 permissions. all the
files in it
should also be owned by postgres.

also look and see if postgres logged anything in its
own system
log files (/var/log/postgresql/9.3 or whatever).

--
john r pierce, recycling bits in santa cruz

--
Sent via pgsql-general mailing list
(pgsql-general@postgresql.org
<mailto:pgsql-general@postgresql.org>
<mailto:pgsql-general@postgresql.org
<mailto:pgsql-general@postgresql.org>>)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/04/2015 02:45 AM, Paolo De Michele wrote:

Adrian,

thanks a lot
about your questions

1. postgresql was installed via apt-get install postgresql -y command
2. in /var/run/postgresql/ there is only one file called 9.3-main.pid
3. postgres is not running, I did not find nothing with ps command
4. I remove the postmaster.pid file and I tried to run postgresql
manually. this is the result:

# /etc/init.d/postgresql start
* Starting PostgreSQL 9.3 database server
The PostgreSQL server failed to start. Please check the log output.
...fail!

Also meant to ask, do you still have the supervisor task running with?:

autostart=true

If so I would suggest changing that to:

autostart=false

while you are troubleshooting this.

I verified in /var/log/postgresql/ and I've not found nothing

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general