Pgsql troubleshooting & Iscsi

On 02/17/2016 09:07 AM, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a logical volume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I don't lose any data, database is available)

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service iscsi.service" etc... but the result is the same : failure in starting postgresql

systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql

^ File system not mounted yet.

systemd: starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

^ Cannot find mount point

kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql

^ Now file system is mounted.

Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

Find where file system mounting is completed and start Postgres after
that. I am still figuring out systemd, so I can not offer anything more
concrete.

regards,
Magique

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 02/17/2016 06:07 PM, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a logical volume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I don't lose any data, database is available)

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service iscsi.service" etc... but the result is the same : failure in starting postgresql

systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql
systemd: starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql
Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

regards,
Magique

Put _netdev in the option field in fstab. This indicates that the
device must be mounted after network is available.

I have for example :

/dev/disk/by-path/ip-10.48.4.1:3260-iscsi-iqn.2004-04.com.qnap:ts-851:iscsi.lun0.e6ae62-lun-0-part1
/data3 btrfs _netdev 0 0

Regards
Jeroen

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

I already try to change the dependencies with postgresql.service (adding as a prerequesite lvm scan, iscsi) .
The mounting began before postgresql but the mount was successfull after the failure.
----- Mail d'origine -----
De: Adrian Klaver <adrian.klaver@aklaver.com>
À: proj@free.fr, pgsql-general@postgresql.org
Envoyé: Wed, 17 Feb 2016 18:35:15 +0100 (CET)
Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi

On 02/17/2016 09:07 AM, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a logical volume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I don't lose any data, database is available)

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service iscsi.service" etc... but the result is the same : failure in starting postgresql

systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql

^ File system not mounted yet.

systemd: starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

^ Cannot find mount point

kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql

^ Now file system is mounted.

Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

Find where file system mounting is completed and start Postgres after
that. I am still figuring out systemd, so I can not offer anything more
concrete.

regards,
Magique

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 02/17/2016 10:30 AM, proj@free.fr wrote:

I already try to change the dependencies with postgresql.service (adding as a prerequesite lvm scan, iscsi) .
The mounting began before postgresql but the mount was successfull after the failure.

Exactly, the drive was not mounted when the Postgres service started, so
Postgres could not find its data directory. From your log:

postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

----- Mail d'origine -----
De: Adrian Klaver <adrian.klaver@aklaver.com>
À: proj@free.fr, pgsql-general@postgresql.org
Envoyé: Wed, 17 Feb 2016 18:35:15 +0100 (CET)
Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi

On 02/17/2016 09:07 AM, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a logical volume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I don't lose any data, database is available)

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service iscsi.service" etc... but the result is the same : failure in starting postgresql

systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql

^ File system not mounted yet.

systemd: starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

^ Cannot find mount point

kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql

^ Now file system is mounted.

Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

Find where file system mounting is completed and start Postgres after
that. I am still figuring out systemd, so I can not offer anything more
concrete.

regards,
Magique

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

I already mount with the netdev option.
The network service is started before postgresql but not lvm for.
I tried to start lvm service as dependency but no changes.

----- Mail d'origine -----
De: Jeroen van Iddekinge <iddekingej@lycos.com>
À: proj@free.fr, pgsql-general@postgresql.org
Envoyé: Wed, 17 Feb 2016 18:51:42 +0100 (CET)
Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi

On 02/17/2016 06:07 PM, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a logical volume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I don't lose any data, database is available)

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service iscsi.service" etc... but the result is the same : failure in starting postgresql

systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql
systemd: starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql
Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

regards,
Magique

Put _netdev in the option field in fstab. This indicates that the
device must be mounted after network is available.

I have for example :

/dev/disk/by-path/ip-10.48.4.1:3260-iscsi-iqn.2004-04.com.qnap:ts-851:iscsi.lun0.e6ae62-lun-0-part1
/data3 btrfs _netdev 0 0

Regards
Jeroen

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Wed, Feb 17, 2016 at 06:07:29PM +0100, proj@free.fr wrote:

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the
database on an ISCSI device (LUN) inside a logical volume, mounted at
starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
kernel sdv: unknown partition table
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.

When I'm logged on the server, if it try to start manually the database :
systemctl start postgresql --> OK (and I don't lose any data, database is
available)

I think it's a problem of order in the boot process : network service must
be started, then iscsi, then lvm etc... So I tried to force dependencies
on the /usr/lib/systemd/system/postgresql.service adding
"After=lvm-pgscan.service iscsi.service" etc... but the result is
the same : failure in starting postgresql

Any ideas ?

Take a look at this thread:
http://www.gossamer-threads.com/lists/linux/kernel/1332888

I'm not an XFS guru, but it seems that XFS does a bunch of checks just
on/after mount, so perhaps PgSQL cannot access /var/lib/pgsql/data
immediately.

You could igive it a try forcing some delay in systemd conf file, or
tweaking the PgSQL startup script.

--
Alberto Cabello Sánchez
Universidad de Extremadura

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Hej all,

Sorry for the late answer.
I faced the same problem installing PostgreSQL 9.5.2 server on my RHEL 7.2
server.

I solved it by doing the following.

1. vi /etc/fstab
.
.
.
UUID=19881aa7-699a-41ff-bd65-216e1d3de62c /var/lib/pgsql/9.5 xfs
_netdev 0 0

2. vi /usr/lib/systemd/system/postgresql-9.5.service
[Unit]
Description=PostgreSQL 9.5 database server
After=syslog.target
After=network.target
*After=remote-fs.target*
.
.
.

3. systemctl daemon-reload

Now, my server is able to reboot and startup the db as it has the remote-fs
dependency defined

Best Regards,
Tim.

On Thu, Feb 18, 2016 at 8:32 AM, Alberto Cabello Sánchez <alberto@unex.es>
wrote: